Subroutines and Stacks in x86 Assembly

type

status

date

slug

summary

Prologue stage

The subroutine often starts with a prologue, which is a sequence of instructions responsible for preparing the stack and registers for the subroutine execution. This usually includes:

Setting up the frame pointer (EBP or RBP)：

The subroutine saves the current frame pointer (EBP) by pushing it onto the stack and then sets the frame pointer (EBP) to the current stack pointer (ESP), which currently pointing to the address that stores the old EBP address
This allows for consistent access to local variables (EBP-x) and function arguments (EBP + x), regardless of any changes to the stack pointer(ESP) during the subroutine execution.

Allocating space for local variables：

The subroutine decrements the stack pointer (ESP) to reserve space on the stack for its local variables.
the reserved bytes is usually initialized with a default value 0xcc

Saving the caller-saved registers（保存现场）：

These are the registers that the calling function expects to remain unchanged after the subroutine returns
The subroutine should push these registers onto the stack before using them and restore them before returning.
This ensures the calling function's state is preserved.

Subroutine Execution Stage:

In this stage, the actual instructions within the subroutine are executed. These instructions may involve performing calculations, accessing memory, and calling other subroutines.

Epilogue

Before returning, the subroutine executes an epilogue, which is a sequence of instructions responsible for cleaning up the stack and restoring the registers. This typically includes:

Restoring the caller-saved registers:

The subroutine pops the saved registers from the stack, ensuring the calling function's state is restored.

Restoring the frame pointer:

The subroutine pops the original frame pointer (EBP or RBP) from the stack, effectively discarding any local variable space allocated earlier.

Return (RET):

The subroutine executes the RET instruction, which pops the return address from the stack and updates the instruction pointer (EIP or RIP) to that address.
This transfers control back to the calling function, just after the original CALL instruction.

stack balancing

Following the CALL instruction, the main program logic continues, often with instructions that handle the return value from the subroutine or proceed with further operations based on the subroutine's output.

sample code

In the code snippet below, we explore the execution of a very simple C function, which takes two integer parameters and return the sum to function caller.