Debugger trap in Linux and GDB (Part 1)

While I was learning GDB and debugging procedure in Linux, I tried to write some simple C codes to see how GDB works and handle breakpoints. We already know what happed in windows when we want to debug a program but to refresh our memory, I'll explain a little bit about it.

Suppose that we have written this simple code in C language and we compiled it using gcc in windows.


#include <stdio.h> 

int main(int argc, char ** argv){
     int a = 2;
	printf ("This is just a sample\n");
	printf ("The value of a is %d\n",a);
	return 0;
}

This is a very small code just to demonstrate how Debuggers put breakpoints in code. If you run this code, the output is:

This is just a sample
The value of a is 2

Which we are not interested in. Now let's open our binary in IDA pro (Interactive disassembler) and put some breakpoints in different parts of the code.

This is how our code looks like in IDA:

click here to see the larg image

This is our main function and you can see we have put a breakpoint at address .text:00401350 (the red line). Now go to option menu -> general and change the value of `number of opcode byte` from 0 to 8 and then go to debugger menu -> debugger options and check the `show debugger breakpoint instruction` option and click ok to confirm.

Now if you run the program through debugger, you should see something like this:

click here to see the larg image

As you can see, there is an extra instruction (int 3) with opcode 0xCC injected into the code by debugger. You can see from the Intel software developer's manual that this opcode (0xCC) is intended for calling the debug exception handler and that's exactly why it is only one byte because you can replace any instruction or at least the first byte of any instruction with this value. This instruction called "Trap to Debugger".

Now let's see how it works in Linux using GDB.

Compile the same program in gcc using this command:

gcc -o sample -m32 -g sample.c

(-g option is for debugging purpose and -m32 means compile it in 32-bit mode).

And then run GDB using 'gdb -q sample' command and type 'start' to start the debugging process. GDB automatically put a breakpoint on line four of our code which is 'int a=4;' (you can see the source code in GDB using 'list' command. If we look at the assembly code of our program using /r option like 'disassemble /r main', we see something like this:

click here to see the larg image

The first column is the address of the instructions, the second one is the opcode for that instruction and the third column is the instruction itself and finally the last column is the operands of the instructions. The selected line in the picture is where we are right now and where the breakpoint inserted by GDB but as you can see there is no 0xCC code in its opcode (the instruction opcode is c7 45 f4 02 00 00 00 but we expect to see something like cc 45 f4 02 00 00 00). At first I thought that maybe the 'disassemble /r address' command in GDB gives us the original code not the modified one in the memory, so I tried to use examine (x) command to see the content of the memory which you can see in the image below:

click here to see the larg image

See? Still no 0xCC code in the output. So what is going on here?

Actually, I searched a lot and found nothing about how to see the breakpoints in GDB (which is actually not that important) but somehow I found out that what you can see in GDB outputs is the original code of the program not the one which modified by GDB (this is true only for breakpoints since we know that we could change code using GDB).

So this behavior tempted me to write another code in C to see the real 0xCC code while code execution which is listed below:

 

#include <stdio.h>

int func_one(){
        int a = 2;
        int b = 3;
        char  buffer[] = "This is just a test\n";
        printf("a is %d\n",a);
        printf("b is %d\n",b);
        printf ("buffer is %s\n",buffer);
        return 1;
}

int main(){
        int(*func_ptr)();
        func_ptr = func_one;
        unsigned char * data;
        data = (unsigned char *) func_ptr;
        func_ptr();
        //Looking for a 0xcc (int3) in func_one
        int col_count = 0;
        printf("OPcode of the func_one is: \n");
        while (*data != 0xC3){  //go ahead until ret instruction
                printf("0x%02x ",*data);
                if (*data == 0xCC){ //looking for int3 assembly code which is 0xcc
                        printf("Debugger found....exit\n");
                        return -1;
                }
                if(col_count++ ==10){
                        printf("\n");
                        col_count = 0;
                }
                data += 1;
        }
        printf("\nNo debugger found!!!\n");
        return 0;
}

This is actually a very simple code. The func_one function just defines some local variables and prints them. This is the function we want to set a breakpoint on it. In the main function, I define a pointer to func_one function which called func_ptr and using this pointer we try to print the opcode of the func_one function and if it finds the 0xCC opcode, it will print a message. That's it.

First let's run the program without the presence of the debugger and the output is:

click here to see the larg image

You see that the program prints the func_one opcode the there is no 0xCC as the message says.

Now let's run it using GDB with the command 'gdb -q check_debug' and type 'start' command to start the debugging process.

GDB automatically inserts a breakpoint on line 15 of the code which is 'func_ptr = func_one;'. Now if you compile the program with -g option, you can simply insert a break point at func_one function by typing 'break func_one'.

And then run the program using 'continue' command and this is the new output.

click here to see the larg image

See? There is a 0xCC code in the output and then the program exited. Strange ha?

Anyway, I didn't find out why we can't find the 0xCC instruction in memory but we did find it programmatically. Instead, we learned a very simple and cool anti-debug technique which works exactly the same way in windows operating system.

To further improve our anti-debug technique, I wrote another C program to show you how to handle SIGTRAP (with code 5) in Linux and how to check if debugger present. Take a look at this code:

 


#include  <stdio.h>
#include  <unistd.h>
#include  <signal.h>

int debug_trap = 0;
void signal_handler(int num){
        if(num == 5 ){
                printf("We handled signal %d\n",num);
                debug_trap++;
                return;
        }
}
int func_one(){
        asm volatile ("int $3");
        if(debug_trap == 1){
                return 0;
        }else{
                return -1;
        }
}
int main(){
        signal(SIGTRAP,signal_handler);
        int val = func_one();
        if(val == -1){ //Debugger found
                printf("Debugger found!!!\n");
                return(-1);
        }else{
                printf("No debugger found!!!\n");
                return(0);
        }
        return 0;
}

This is actually a very simple code. We manually execute int 3 instruction using inline assembly and then register a signal handler in our code to handle SIGTRAP signal in our code. If the debugger is present, it will handle the SIGTRAP and as a result, our handler never execute. On the other hand, if there is no debugger, our handler will handle the exception and increment the debug_trap variable to 1. In the main function we check if debug_trap variable is 1 and if so, it means that there is no debugger present. Simple as that!!!

This is the output of the program executing using GDB:

click here to see the larg image

In the next part of this tutorial I'll introduce more anti-debug techniques in Linux environment.