x64 Calling Conventions and Stack Frames

What this is

A reference for the question you hit constantly in reversing and in reading a crash: this function was called with some arguments, and you need argument number four. There are two x64 conventions in common use and they disagree about almost everything — register order, whether the caller reserves stack space for register arguments, and which register anchors local variables.

Note RCX and RDX appear in both lists in different positions. Reading a Linux binary with the Windows order gives you plausible, wrong values — which is the single most common way to misread a frame.

Prerequisites

• A disassembler that shows registers at a breakpoint: WinDBG, Ghidra’s decompiler, GDB, IDA.
• .frame and k in WinDBG; bt and info frame in GDB.
• Knowing which convention applies. It follows the target platform, not the file format — so a Windows PE uses the Microsoft convention even when you are analysing it on Linux.

Walkthrough

Windows x64: the home space is the thing to understand

Four integer arguments go in RCX, RDX, R8, R9. Arguments five and beyond are pushed. The non-obvious part is that the caller must also reserve 32 bytes immediately above the return address — the home space — whether or not the callee uses it, and whether or not the function takes four arguments at all.

Inside a function that has executed its prologue:

RSP + 0x00   return address
RSP + 0x08   home space slot for arg 1  (the RCX slot)
RSP + 0x10   home space slot for arg 2  (the RDX slot)
RSP + 0x18   home space slot for arg 3  (the R8 slot)
RSP + 0x20   home space slot for arg 4  (the R9 slot)
RSP + 0x28   argument 5
RSP + 0x30   argument 6

The home space exists so the callee has somewhere to spill the register arguments, which it does whenever it needs their addresses or needs the registers back. That spill is why you can often recover arguments one through four after the registers have been clobbered: look in the home space.

The other Windows-specific habit: RBP is not a frame pointer. Locals and arguments are addressed RSP + offset, and RSP does not move through the body of the function — the prologue subtracts the whole frame at once and the epilogue adds it back. So an RSP-relative offset is stable anywhere in the body, which is what makes reading these frames mechanical.

In a decompiler this shows up as the first four arguments being read straight out of the registers at the top of the function — often as 32-bit sub-registers such as ECX when the parameter is an int, which is worth noticing because it tells you the declared width.

Reading a frame in WinDBG:

k
.frame 2
dv /V
dq @rsp L8

dv /V prints locals with the register or RSP offset each one lives at, which saves the arithmetic when symbols are present. Without symbols, dq @rsp L8 and the table above is the whole method.

System V: six registers, no home space, RBP-anchored

Six integer registers in the order RDI, RSI, RDX, RCX, R8, R9, then the stack. Nothing is reserved for the register arguments, so the stacked arguments start directly above the saved frame pointer.

With a frame pointer established, the stacked arguments are at fixed positive offsets from RBP:

RBP + 0x00   saved RBP
RBP + 0x08   return address
RBP + 0x10   argument 7
RBP + 0x18   argument 8
RBP + 0x20   argument 9

The spilled-argument offset that moves

Register arguments are often spilled to negative RBP offsets so the function can treat them like locals. Those offsets are not fixed — they depend on how much local space the function allocated, because the compiler lays out locals first.

A four-byte first argument in a function with no locals lands at rbp - 0x4. Add one local and it moves to rbp - 0x14. Cross 16 bytes of locals and it moves to rbp - 0x24; cross 32 and it is at rbp - 0x34. The shift is 0x10 per 16-byte block, which is the alignment the ABI requires.

Two consequences. Do not carry a spilled-argument offset from one function to another, and do not carry one across a recompile. And when you are looking for a value and the obvious offset is wrong, count the locals before assuming you have the wrong register.

At main, the same registers hold the program’s own arguments: RDI is argc and RSI points at argv, whose first element is the program path. Walk further up the stack and you reach the environment block — which is where you read the environment of a process whose /proc entry you cannot trust.

What it tells you

Which byte to read when the symbol is missing. This is the practical payoff. A memory image gives you a stack; a sandbox gives you a call with no prototype. The convention tables above turn “a call happened here” into “argument three was this pointer”, which is what you need to recover a filename, a URL, a buffer length or a key.

How to reconstruct a call you only have the aftermath of. On Windows, the home space frequently still contains the first four arguments after the function has clobbered the registers, because the callee spilled them there. On a frame that has already returned, the values above the old RSP are often intact, since nothing zeroes a stack on return. Reading a stale frame out of a memory image is a standard way to recover an argument whose caller is gone.

Confidence in a hook’s argument indices. Every Interceptor.attach handler in API tracing with Frida indexes args[n], and that index is only correct because of the table at the top of this sheet. When a hook prints nonsense, the first thing to check is whether you are applying the right convention — a Windows API read with System V ordering gives you args[0] and args[1] transposed into the wrong slots and no error anywhere.

Reading the stack trace in an injection investigation. A backtrace that crosses from a non-image-backed region into kernel32 or ntdll is the shape of injected code calling the OS, and the frame below the boundary tells you what it asked for. That is the step between “injected-thread hunting flagged this thread” and “this thread opened that file”. It needs frame-walking, which needs the convention.

Alignment as a sanity check. Both ABIs require RSP to be 16-byte aligned at a call instruction. A frame where RSP is misaligned is either mid-prologue, hand-written assembly, or you have the wrong frame — and shellcode that ignores alignment is a common source of crashes in otherwise working injection, which is worth knowing when a payload faults immediately on entry.

Varargs are special on System V. A variadic call sets AL to the number of vector registers used. If you are reading arguments to a printf-family function and the values are wrong, that is usually why.

References

• x64 stack usage — Microsoft Learn
• x64 calling convention — Microsoft Learn
• System V AMD64 ABI
• API tracing with Frida — where the argument indices get used in anger