Basics

Intro to error handling and some basic techniques

Since this blog is not about Windows system programming, i will not be covering C or Win API basics. this section is about some basic tricks and considerations to make your life easier while dealing with Win API crazy syntax, structures, data types and error codes.

---

Error Handling

GetLastError

This is the most used Win API for resolving error codes. it checks the return code of the function/sub-routine that was executed right before and returns an integer code which can be checked with VisualStudio "Error Lookup" tool from the "Tools" menu.

GetLastError function (errhandlingapi.h) - Win32 appsMicrosoftLearn

Here is an example:

GetLastError.c

#include <windows.h>
#include <stdio.h>

int main() {
	// Attempt to open a non-existent file
	HANDLE hFile = CreateFile(
		"non_existent_file.txt",   // File name
		GENERIC_READ,              // Desired access
		0,                         // Share mode
		NULL,                      // Security attributes
		OPEN_EXISTING,            // Creation disposition
		FILE_ATTRIBUTE_NORMAL,     // File attributes
		NULL                       // Template file
	);

	// Check if the file handle is invalid
	if (hFile == INVALID_HANDLE_VALUE) {
		// Retrieve the last error code
		DWORD dwError = GetLastError();

		// Print the error code and a message
		printf("Failed to open file. Error code: %lu\n", dwError);
	}
	else {
		// If successful, close the file handle
		CloseHandle(hFile);
	}

	return 0;
}

output:

Failed to open file. Error code: 2

After looking up the error code, we can find the error message:

---

GetLastErrorAsString

Going a step further in error look up techniques, there is a function called FormatMessage in Win API that turns the error code into a human-readable string (the string is exactly the same as error lookup tool).

GetLastErrorAsString.c

#include <windows.h>
#include <stdio.h>

void GetLastErrorAsString(DWORD dwError) {
    LPVOID lpMsgBuf;

    // Format the error message from the error code
    FormatMessage(
        FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
        NULL,
        dwError,
        0, // Default language
        (LPWSTR)&lpMsgBuf,
        0,
        NULL
    );

    // Display the error message
    wprintf(L"Error code %lu: %s", dwError, (LPWSTR)lpMsgBuf);

    // Free the buffer allocated by FormatMessage
    LocalFree(lpMsgBuf);
}

int main() {
    // Attempt to open a non-existent file
    HANDLE hFile = CreateFile(
        "non_existent_file.txt",   // File name
        GENERIC_READ,              // Desired access
        0,                         // Share mode
        NULL,                      // Security attributes
        OPEN_EXISTING,             // Creation disposition
        FILE_ATTRIBUTE_NORMAL,     // File attributes
        NULL                       // Template file
    );

    // Check if the file handle is invalid
    if (hFile == INVALID_HANDLE_VALUE) {
        // Retrieve the last error code
        DWORD dwError = GetLastError();

        // Print the error message as a string
        GetLastErrorAsString(dwError);
    }
    else {
        // If successful, close the file handle
        CloseHandle(hFile);
    }

    return 0;
}

output:

Error code 2: The system cannot find the file specified.

---

SetLastError

In some cases, you might need to set the return code of a function to check it later in the code. for example, lets say you wrote a custom function or you want to change all error codes to a custom category (using Win API macros or intigers).

SetLastError function (errhandlingapi.h) - Win32 appsMicrosoftLearn

example:

SetLastError.c

#include <windows.h>
#include <stdio.h>

int main() {
    // Set a custom error code using SetLastError
    SetLastError(ERROR_ACCESS_DENIED);  // Error code 5: Access Denied

    // Retrieve the error code using GetLastError (to verify it was set)
    DWORD dwError = GetLastError();

    SetLastError(1234);  // Error code 5: Access Denied

    // Retrieve the error code using GetLastError (to verify it was set)
    DWORD dwError2 = GetLastError();

    // Print the error code that was set
    printf("Error code set by SetLastError: %lu\n", dwError);
    printf("Error code set by SetLastError: %lu\n", dwError2);

    return 0;
}

output:

Error code set by SetLastError: 5
Error code set by SetLastError: 1234

---

Debug Print

Using CRT "printf"

This really doesn't need an explanation, we all use print statements to save our asses while debugging code without a debugger. here is a simple macro that can be used for that, it also has a compiler directive to enable/disable print statements across the entire code by simply commenting/un-commenting a single line.

crt_debug_print.c

#include <stdio.h>
#include <windows.h>
#include <stdarg.h>   // Include standard arguments library

// debug print toggle, comment this to disable debug print statements !!!!
#define DEBUG_MODE

// Function prototype for printd that accepts a format string and variable arguments
void printd(const char* format, ...) {
#ifdef DEBUG_MODE              // if DEBUG_PRINT is defined, enable printing
    va_list args;              // Declare a variable of type va_list to hold the argument list
    va_start(args, format);    // Initialize the va_list with the last fixed argument (format)
    // Print the formatted output using vprintf, which takes the format string and va_list
    vprintf(format, args);
    // Clean up the va_list after use
    va_end(args);
#endif
}

int main(int argc, char* argv[]) {
    printd("Integer: %d, String: %s, Character: %c\n", 42, "test", 'A');
    return 0;
}

when #define DEBUG_MODE is not commented, the output will be like this:

Integer: 42, String: test, Character: A

If we comment that line, printd function will return without doing anything and all debug print statements will be disabled.

---

Using Win32 API

Previous code was good enough for debugging when our code is using standard C run-time (CRT) libraries. but in some cases (discussed in later posts), we want to completly strip out CRT and only use Win32 API or our own custom library code.

In such cases, we can use the following code which is an implementation of CRT "printf" with mode tuggle and support for integers, strings, pointers and characters.

win_debug_print.c

#include <windows.h>

// Debug print toggle, comment this to disable debug print statements
#define DEBUG_PRINT

// Function to write a string to the console
void write_to_console(const char* str, int length) {
    HANDLE hConsole = GetStdHandle(STD_OUTPUT_HANDLE);
    DWORD written;
    WriteConsoleA(hConsole, str, length, &written, NULL);
}

// Custom function to convert integer to string
void my_itoa(int value, char* str, int base) {
    int i = 0;
    int isNegative = 0;

    // Handle 0 explicitly
    if (value == 0) {
        str[i++] = '0';
        str[i] = '\0';
        return;
    }

    // Handle negative integers
    if (value < 0 && base == 10) {
        isNegative = 1;
        value = -value;
    }

    // Process individual digits
    while (value != 0) {
        int remainder = value % base;
        str[i++] = (remainder > 9) ? (remainder - 10) + 'a' : remainder + '0';
        value = value / base;
    }

    // Append negative sign for negative numbers
    if (isNegative) {
        str[i++] = '-';
    }

    // Null-terminate the string
    str[i] = '\0';

    // Reverse the string
    for (int j = 0; j < i / 2; j++) {
        char temp = str[j];
        str[j] = str[i - j - 1];
        str[i - j - 1] = temp;
    }
}

// Custom function to convert pointer to hexadecimal string
void my_ptrtoa(void* ptr, char* str, int max_size) {
    unsigned long long value = (unsigned long long)ptr;
    int i = 0;

    if (max_size < 3) {
        return;
    }

    if (value == 0) {
        str[i++] = '0';
    }
    else {
        while (value != 0 && i < max_size - 1) {
            int remainder = value % 16;
            str[i++] = (remainder > 9) ? (remainder - 10) + 'a' : remainder + '0';
            value = value / 16;
        }
    }

    for (int j = 0; j < i / 2; j++) {
        char temp = str[j];
        str[j] = str[i - j - 1];
        str[i - j - 1] = temp;
    }

    str[i] = '\0';
}

// Print function that accepts a format string and various arguments
void printd(const char* format, ...) {
#ifdef DEBUG_PRINT
    char buffer[256];
    char temp[64]; // Temporary buffer for formatted output
    int i = 0, j = 0;

    va_list args;
    va_start(args, format); // Initialize va_list

    while (format[i] != '\0') {
        if (format[i] == '%') {
            i++;
            if (format[i] == 's') { // Handle string
                const char* str = va_arg(args, const char*);
                while (*str != '\0' && j < sizeof(buffer) - 1) {
                    buffer[j++] = *str++;
                }
            }
            else if (format[i] == 'd') { // Handle integer
                int num = va_arg(args, int);
                my_itoa(num, temp, 10);
                for (int k = 0; temp[k] != '\0'; k++) {
                    buffer[j++] = temp[k];
                }
            }
            else if (format[i] == 'c') { // Handle character
                char ch = (char)va_arg(args, int);
                buffer[j++] = ch;
            }
            else if (format[i] == 'p') { // Handle pointer
                void* ptr = va_arg(args, void*);
                my_ptrtoa(ptr, temp, sizeof(temp));
                for (int k = 0; temp[k] != '\0'; k++) {
                    buffer[j++] = temp[k];
                }
            }
            else {
                // Handle unknown specifier
                buffer[j++] = '%';
                buffer[j++] = format[i];
            }
        }
        else {
            buffer[j++] = format[i];
        }
        i++;
    }

    buffer[j] = '\0';
    write_to_console(buffer, j);
    va_end(args); // Clean up va_list
#endif
}

// Testing printd function
int main() {
    char a[] = "test";
    int num = 42;
    char ch = 'A';
    printd("String is: %s\nInteger is: %d\nCharacter is: %c\nPointer is: 0x%p\n", a, num, ch, &num);
    printd("this is a test");
    return 0;
}

Output:

String is: test
Integer is: 42
Character is: A
Pointer is: e0fd3afaf4
this is a test

---

Hiding the Console

After you`re done with the code, your malware should silently execute on victim machine, so you probably don't want the nasty cmd shell popup after execution. there are a couple of tricks to hide the cmd shell console.

Hiding the Console Window After It’s Created

If you want to hide the console window after it is created, you can use the ShowWindow function to hide the console.

ShowWindow.c

#include <windows.h>
#include <stdio.h>

int main() {
    // Retrieve the current console window handle
    HWND hWnd = GetConsoleWindow();

    // Hide the console window
    ShowWindow(hWnd, SW_HIDE);

    // Your code here (the console is hidden)
    MessageBox(NULL, L"The console is hidden", L"Hidden Console", MB_OK);

    return 0;
}

After execution, the console will not be visible, even though MessageBox works correctly:

Free the Console (Detach the Console)

We can also free the console from your process using the FreeConsole function, which detaches the console window from your application.

FreeConsole.c

#include <windows.h>
#include <stdio.h>

int main() {
    // Detach the console from the current process
    FreeConsole();

    // Your code here (the console is no longer attached)
    MessageBox(NULL, "The console is detached", "Detached Console", MB_OK);

    return 0;
}

In this case, the console window is closed and detached from the process after the call to FreeConsole.

Prevent Console Creation in the First Place

One way to avoid showing a console window is to create a Windows application (rather than a console application) by using the WinMain entry point instead of main.

WinMain.c

#include <windows.h>

int APIENTRY WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nCmdShow) {
    // Your hidden application code here
    MessageBox(NULL, "The console is hidden", "Hidden Console", MB_OK);
    return 0;
}

In this case, no console window is created, as WinMain is used for GUI applications, which don’t automatically open a console window.

Changing Subsystem After Compilation

In case you are dealing with a sample or tool that is already compiled, you can modify the PE to change the subsystem from CLI to GUI. this acts exactly like the previous technique and hides the console.

To do this, first open the PE file in DetectItEasy Tool:

GitHub - horsicq/Detect-It-Easy: Program for determining types of files for Windows, Linux and MacOS.GitHub

Check "Advanced" option and go to "File Info" :

In the next window, uncheck "Readonly" :

Then go to IMAGE_NT_HEADERS > IMAGE_OPTIONAL_HEADER from the left panel and change the subsystem DWORD value to WINDOWS_GUI.

Save and exit, now the console should be gone.

Ignore the prompt, the original file gets modified.