EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models (2018-04-12T00:00:00.000000Z)

TL;DR

The authors hope that the dataset, code and baseline model provided by EMBER will help invigorate machine learning research for malware detection, in much the same way that benchmark datasets have advanced computer vision research.

Abstract

This paper describes EMBER: a labeled benchmark dataset for training machine learning models to statically detect malicious Windows portable executable files. The dataset includes features extracted from 1.1M binary files: 900K training samples (300K malicious, 300K benign, 300K unlabeled) and 200K test samples (100K malicious, 100K benign). To accompany the dataset, we also release open source code for extracting features from additional binaries so that additional sample features can be appended to the dataset. This dataset fills a void in the information security machine learning community: a benign/malicious dataset that is large, open and general enough to cover several interesting use cases. We enumerate several use cases that we considered when structuring the dataset. Additionally, we demonstrate one use case wherein we compare a baseline gradient boosted decision tree model trained using LightGBM with default settings to MalConv, a recently published end-to-end (featureless) deep learning model for malware detection. Results show that even without hyper-parameter optimization, the baseline EMBER model outperforms MalConv. The authors hope that the dataset, code and baseline model provided by EMBER will help invigorate machine learning research for malware detection, in much the same way that benchmark datasets have advanced computer vision research.

Authors

H. Anderson

1 papers

Phil Roth

1 papers

TL;DR

Abstract

Authors

References29 items

Microsoft Malware Classification Challenge

LightGBM: A Highly Efficient Gradient Boosting Decision Tree

Malware Detection by Eating a Whole EXE

Malware classification with LSTM and GRU language models and a character-level CNN

Deep neural network based malware detection using two dimensional binary program features

Malware classification with recurrent networks

Detection of traffic signs in real-world images: The German traffic sign detection benchmark

Large-scale malware classification using random projections and neural networks

ImageNet classification with deep convolutional neural networks

Scikit-learn: Machine Learning in Python

Malware Forensics: Discovery of the Intent of Deception

PE-Miner: Mining Structural Information to Detect Malicious Executables in Realtime

ImageNet: A large-scale hierarchical image database

Feature hashing for large scale multitask learning

Learning to detect malicious executables in the wild

Data mining methods for detection of new malicious executables

Fast Effective Rule Induction

Speech database development at MIT: Timit and beyond

Computer viruses: Theory and experiments

How to build a malware classifier

PE File Features in Detection of Packed Executables

Selecting Features to Classify Malware

Amazon's mechanical turk: A new source of inexpensive, yet high-quality, data? Perspectives on psychological science

A Framework for Efficient Mining of Structural Information to Detect Zero-Day Malicious Portable Executables

Twitter Sentiment Classiﬁcation using Distant Supervision

Inside windows-an in-depth look into the win32 portable executable file format. MSDN magazine

Gradient-based learning applied to document recognition

Virustotal-free online virus, malware and url scanner

: library for instrumenting executable files

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names