Alexandria Digital Research Library

Scalable automated evasive malware analysis

Kirat, Dhilung Hang
Degree Grantor:
University of California, Santa Barbara. Computer Science
Degree Supervisor:
Giovanni Vigna and Christopher Kruegel
Place of Publication:
[Santa Barbara, Calif.]
University of California, Santa Barbara
Creation Date:
Issued Date:
Computer science
Computer security
Evasive malware
Malware analysis
Signal processing
Bare metal
Dissertations, Academic and Online resources
Ph.D.--University of California, Santa Barbara, 2015

The threat landscape of malicious applications, or malware, is persistently growing and evolving. Malware has become one of the major offensive components of the global cybersecurity threat. Accurate understanding of malware behavior is a crucial step towards developing systems that deter, detect, and defend against malware threats. Unfortunately, the widely deployed signature-based and lightweight static-analysis-based detection techniques (Antivirus) are easily evaded by techniques commonly seen in the wild, such as code obfuscation, packing, and encryption. Recent malware detection systems are moving towards a more robust dynamic analysis approach. These systems execute suspicious samples in a controlled environment, called "sandbox", and observe malicious intent through their dynamic behavior. However, many sophisticated evasive malware samples are evading such analysis by first detecting the analysis environment and then stopping their malicious activities. Because of the sophisticated and evolving techniques used by the malware authors, so far the analysis and detection of evasive malware has been largely a manual process. This manual approach is not scalable to tens of thousands of new malware samples that we observe every day.

In this dissertation, I will present my research on scalable and automated evasive malware analysis. First, I will discuss my work on reducing the input load to a dynamic analyzer and present a static-filter called SIGMAL, which is based on image processing and machine learning algorithms. I will outline a bare metal-based approach to evasive malware analysis. Then, I will describe BAREBOX, a novel technique for improving the scalability of this approach. Next, I will propose a new algorithm for detecting deviation in the malware behaviors among different execution environments. The BARECLOUD project leverages this algorithm and builds a system for automatically detecting realworld evasive malware at a large-scale. Finally, I will present MALGENE, a first step towards the automatic extraction of evasion signatures from evasive malware samples. Given millions of incoming suspicious samples as a daily input, SIGMAL can quickly identify newer and relevant samples, BAREBOX can execute them transparently in a bare metal environment and generate behavior profiles, which BARECLOUD can use to automatically detect evasive samples. Given these evasive samples, MALGENE can automatically extract evasion signatures and cluster them according to the underlying evasion techniques.

Physical Description:
1 online resource (218 pages)
UCSB electronic theses and dissertations
Catalog System Number:
Inc.icon only.dark In Copyright
Copyright Holder:
Dhilung Kirat
Access: This item is restricted to on-campus access only. Please check our FAQs or contact UCSB Library staff if you need additional assistance.