"In computing, a system call (commonly abbreviated to syscall) is the programmatic way in which a computer program requests a service from the operating system on which it is executed." - Wikipedia
I've recently been working on a new cheat and it's become necessary to take steps to obscure my code. To do this, I've taken on the task of learning to run syscalls myself rather than relying on the potentially hooked exports from NtDll.
One of the ways Anticheats can flag a cheater is by monitoring calls to NtDll functions such as NtAllocateVirtualMemory. They often do this through inline hooks on those functions, but there are many approaches that can be taken. When a cheat calls a function hooked by the anticheat, it can use an arsenal of techniques to determine if the caller appears nefarious.
A simple way of avoiding those hooks is simply not calling a hooked function, but when those hooks are on functions which talk to the kernel, the only alternative is to talk to the kernel yourself.
In my case, I need the functionality contained within the following NtDll routines:
The kernel identifies the syscalls these routines make by an integer identifier. These identifiers change between windows versions, so I needed to find the correct IDs for my system. To do this, I popped NtDll into IDA Pro and looked over the disassembly.
Here is NtAllocateVirtualMemory:
We can see line #2 sets the value of the ECX register to 24, this tells us the syscall ID for NtAllocateVirtualMemory is 24.
Knowing the x64 calling convention, I found line #1 curious. It sets the value of r10 to rcx. This is because the syscall operation destroys the RCX register, and thus cannot be used to pass the 1st argument to the kernel; r10 takes this role instead.
The first step in doing manual syscalls is to write my own routine for the syscall operation. I wanted this routine to be used for any syscall, so I needed to set the value of ecx dynamically.
This routine takes in the syscall ID on the r12d register, otherwise it's no different than the NtDll routines. Next I am going to create functions for each syscall I want to replicate. Here is NtQueryVirtualMemory:
This function takes 5 arguments, one for the Syscall ID, and 4 which pass into the 6 argument NtQueryVirtualMemory syscall.
One of the important things to note is that the shadow space still needs allocated (L26-29) (Note: I call it the "shadow pool" in reference to Dota 2 smurfing).
With this, we can jump back into C and finish writing a nice wrapper function for our syscall. We've successfully created a function which executes a syscall manually.
The next step would be making this syscall work across multiple versions of Windows. A simple approach to this would be to read NtDll from disk and look for the syscall IDs before calling any of them. Here is an example of how that could be done:
Unfortunately, manual syscalls are not a foolproof method. There exists many ways of detecting these, such as instrumentation callbacks, which make running a syscall operation from outside NtDll very suspicious.
My novel (and naïve) approach to tackle this was to modify my syscall_idx routine such that it JUMPs into NtDll to perform the syscall operation. That looks a little something like:
Before making a syscall, I find a syscall; ret; instruction pair in NtDll and call set_nt_syscall_addr(found_address); to ensure syscall_idx knows where to jump.
This isn't flawless. An Anticheat could simply check the return address to see where the ret instruction inside NtDll is returning to (hint: it returns to my cheat).
...
Anyway it was a fun learning exercise & definitely helped me learn more about user<->kernel interaction.