In-Memory ShellCode Detection Using a Patterns-Based Methodology
During an analysis, it can be really useful to know some common instructions with which malware, and more specifically shellcodes, achieve their goals. As we can imagine, these sets of common instructions could be used first to locate and later to analyze and/or to identify general threats: embedded or injected code.
In this article, we’ll focus on the identification and analysis of Metasploit and some custom shellcodes on the basis of parameters and information coming from brief research and personal experience.
We’ll start our analysis from some previously created memory images. Some of these come directly from real incidents that have occurred, while others are specially created.
We’ll look at several shellcodes in order to understand exactly how they work and how to recognize them on the basis of characteristics you notice during the analysis of a possible incident, and/or on the basis of instructions that in most cases are performed to make the shellcode as reusable and effective as possible.
It’s important however to understand that this is not an exact science and that the techniques used here could work for some situations, but could not be useful for others. Bad guys put in place many techniques to make their code as stealthy as possible, like the use of many alternative instructions to achieve the same result, the use of unnecessary instructions (push reg, pop reg, xor reg, reg) to avoid pattern recognition, and obviously the encoding of payloads (also used to avoid null bytes).
In general, it’s always necessary that you put into play your own experience and knowledge to deal with a wide range of situations.
This article has no pretensions to cover every possible eventuality in the identification of malicious shellcodes (just because of the concepts expressed above, this would be impossible), but presents an approach based on the creation of recognition of patterns based on a particular behavior of a service (such as listening port to a non-conventional) and/or on the research for those operations that shellcode must necessarily run to be effective (such as the recovery of the IP, since with the instruction ‘mov eax, EIP’ it’s still not valid :] ).