Learning the PE Header, Malware Detection with Minimal Domain Knowledge

Published in

(2017)

External Links:

Generate Graph DownloadPDF

TL;DR

This work explores the feasibility of applying neural networks to malware detection and feature learning by restricting ourselves to a minimal amount of domain knowledge in order to extract a portion of the Portable Executable (PE) header.

Abstract

Many efforts have been made to use various forms of domain knowledge in malware detection. Currently there exist two common approaches to malware detection without domain knowledge, namely byte n-grams and strings. In this work we explore the feasibility of applying neural networks to malware detection and feature learning. We do this by restricting ourselves to a minimal amount of domain knowledge in order to extract a portion of the Portable Executable (PE) header. By doing this we show that neural networks can learn from raw bytes without explicit feature construction, and perform even better than a domain knowledge approach that parses the PE header into explicit features.

Authors

Edward Raff

7 papers

Jared Sylvester

2 papers

Charles K. Nicholas

3 papers

References71 items

An investigation of byte n-gram features for malware classification

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

The mythos of model interpretability

Structural analysis of binary executable headers for malware detection optimization

“Why Should I Trust You?”: Explaining the Predictions of Any Classifier

Learning the PE Header, Malware Detection with Minimal Domain Knowledge

Published in

(2017)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Edward Raff

7 papers

Jared Sylvester

2 papers

Charles K. Nicholas

3 papers

References71 items

An investigation of byte n-gram features for malware classification

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

The mythos of model interpretability

Structural analysis of binary executable headers for malware detection optimization

“Why Should I Trust You?”: Explaining the Predictions of Any Classifier

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Deep Residual Learning for Image Recognition

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Reducing Overfitting in Deep Networks by Decorrelating Representations

Reasoning about Entailment with Neural Attention

Character-level Convolutional Networks for Text Classification

Recognizing Functions in Binaries with Neural Networks

Malware classification with recurrent networks

Describing Videos by Exploiting Temporal Structure

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

How transferable are features in deep neural networks?

Understanding variable importances in forests of randomized trees

Visualizing and Understanding Convolutional Networks

Generating Sequences With Recurrent Neural Networks

Classification of malware families based on N-grams sequential pattern features

Efficient Estimation of Word Representations in Vector Space

ImageNet classification with deep convolutional neural networks

On the difficulty of training recurrent neural networks

Building high-level features using large scale unsupervised learning

An improved GLMNET for l1-regularized logistic regression

The Balanced Accuracy and Its Posterior Distribution

Rectified Linear Units Improve Restricted Boltzmann Machines

Why Does Unsupervised Pre-training Help Deep Learning?

Regularization Paths for Generalized Linear Models via Coordinate Descent.

PE-Miner: Mining Structural Information to Detect Malicious Executables in Realtime

Unknown malcode detection and the imbalance problem

Detection of malicious code by applying machine learning classifiers on static features: A state-of-the-art survey

Improving malware detection by applying multi-inducer ensemble

McBoost: Boosting Scalability in Malware Collection and Analysis Using Statistical Classification of Executables

Unknown Malcode Detection Using OPCODE Representation

Cloud-based malware detection for evolving data streams

Embedded Malware Detection Using Markov n-Grams

A scalable multi-level feature extraction technique to detect malicious executables

Applying Machine Learning Techniques for Detection of Malicious Code in Network Traffic

Malicious Codes Detection Based on Ensemble Learning

Learning to Detect and Classify Malicious Executables in the Wild

Hunting for metamorphic engines

Extremely randomized trees

Addendum: Regularization and variable selection via the elastic net

Predicting good probabilities with supervised learning

What is dynamic programming?

Learning to Forget

AUC Optimization vs. Error Rate Minimization

A Neural Probabilistic Language Model

Transforming classifier scores into accurate multiclass probability estimates

Data mining methods for detection of new malicious executables

On the Surprising Behavior of Distance Metrics in High Dimensional Spaces

Learning to Forget: Continual Prediction with LSTM

Long Short-Term Memory

JSAT: Java Statistical Analysis Tool, a Library for Machine Learning

Virus share. url: https

Robust Static Analysis of Portable Executable Malware

Dropout: a simple way to prevent neural networks from overfitting

Microsoft Portable Executable and Common Object File Format Speci cation Version 8.3

Detection of Intrusions and Malware, and Vulnerability Assessment

Detection of Intrusions and Malware, and Vulnerability Assessment, 5th International Conference, DIMVA 2008, Paris, France, July 10-11, 2008. Proceedings

TL;DR

Abstract

Authors

References71 items

An investigation of byte n-gram features for malware classification

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

The mythos of model interpretability

Structural analysis of binary executable headers for malware detection optimization

“Why Should I Trust You?”: Explaining the Predictions of Any Classifier

TL;DR

Abstract

Authors

References71 items

An investigation of byte n-gram features for malware classification

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

The mythos of model interpretability

Structural analysis of binary executable headers for malware detection optimization

“Why Should I Trust You?”: Explaining the Predictions of Any Classifier

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Deep Residual Learning for Image Recognition

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Reducing Overfitting in Deep Networks by Decorrelating Representations

Reasoning about Entailment with Neural Attention

Character-level Convolutional Networks for Text Classification

Recognizing Functions in Binaries with Neural Networks

Malware classification with recurrent networks

Describing Videos by Exploiting Temporal Structure

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

How transferable are features in deep neural networks?

Understanding variable importances in forests of randomized trees

Visualizing and Understanding Convolutional Networks

Generating Sequences With Recurrent Neural Networks

Classification of malware families based on N-grams sequential pattern features

Efficient Estimation of Word Representations in Vector Space

ImageNet classification with deep convolutional neural networks

On the difficulty of training recurrent neural networks

Building high-level features using large scale unsupervised learning

An improved GLMNET for l1-regularized logistic regression

The Balanced Accuracy and Its Posterior Distribution

Rectified Linear Units Improve Restricted Boltzmann Machines

Why Does Unsupervised Pre-training Help Deep Learning?

Regularization Paths for Generalized Linear Models via Coordinate Descent.

PE-Miner: Mining Structural Information to Detect Malicious Executables in Realtime

Unknown malcode detection and the imbalance problem

Detection of malicious code by applying machine learning classifiers on static features: A state-of-the-art survey

Improving malware detection by applying multi-inducer ensemble

McBoost: Boosting Scalability in Malware Collection and Analysis Using Statistical Classification of Executables

Unknown Malcode Detection Using OPCODE Representation

Cloud-based malware detection for evolving data streams

Embedded Malware Detection Using Markov n-Grams

A scalable multi-level feature extraction technique to detect malicious executables

Applying Machine Learning Techniques for Detection of Malicious Code in Network Traffic

Malicious Codes Detection Based on Ensemble Learning

Learning to Detect and Classify Malicious Executables in the Wild

Hunting for metamorphic engines

Extremely randomized trees

Addendum: Regularization and variable selection via the elastic net

Predicting good probabilities with supervised learning

What is dynamic programming?

Learning to Forget

AUC Optimization vs. Error Rate Minimization

A Neural Probabilistic Language Model

Transforming classifier scores into accurate multiclass probability estimates

Data mining methods for detection of new malicious executables

On the Surprising Behavior of Distance Metrics in High Dimensional Spaces

Learning to Forget: Continual Prediction with LSTM

Long Short-Term Memory

JSAT: Java Statistical Analysis Tool, a Library for Machine Learning

Virus share. url: https

Deep Learning

Robust Static Analysis of Portable Executable Malware

Dropout: a simple way to prevent neural networks from overfitting

Microsoft Portable Executable and Common Object File Format Speci cation Version 8.3

Detection of Intrusions and Malware, and Vulnerability Assessment

Open Malware

Detection of Intrusions and Malware, and Vulnerability Assessment, 5th International Conference, DIMVA 2008, Paris, France, July 10-11, 2008. Proceedings

Towards Stealthy Malware Detection

Manual on setting up, using, and understanding random forests v4.0

Random Forests