First things first, this won’t be your typical “Use ltrace to solve the challenge” walkthrough. While ltrace is a great tool for reverse engineering, I figured since I haven’t found any write-ups giving a detailed breakdown of how to solve this challenge, I wanted to give a deep dive into how this binary works. In this walkthrough you will learn how to:

• Use Radare2.
• Use Ghidra.
• Read x86 Assembly.
• Calculate addresses on the stack.
• Read data on the registers.
• Tell the difference between Endianness.
• What a bitwise XOR operation is and how to identify it.
• Perform a bitwise XOR operation manually.

I hope you all will learn something valuable reading through this walkthrough.

Happy Hacking!

Static Analysis

As per usual, I start off with pulling strings (using the strings command) and disassembling the binary in Ghidra.

Step 1: Launch Ghidra and Create a New Project

1. Once Ghidra opens, click on File > New Project.
2. Choose Non-Shared Project and click Next.
3. Select a directory to save the project and provide a name for your project.
4. Click Finish to create your project.

Step 2: Import the Binary File

1. With your new project open, click on File > Import File.
2. Browse to the location of the binary file you want to analyze, select it, and click OK.
3. Ghidra will analyze the file automatically, but you can customize the analysis options before proceeding.
4. Click Yes if prompted to perform an initial analysis, or select Auto Analysis later.

Navigating to the entry point in Ghidra, we can make our way to the main function (FUN_0010080a) which is called by __libc_start_main

void processEntry entry(undefined8 param_1,undefined8 param_2)

{
  undefined auStack_8 [8];
  
  __libc_start_main(FUN_0010080a,param_2,&stack0x00000008,FUN_001009c0,FUN_00100a30,param_1,
                    auStack_8);
  do {
                    /* WARNING: Do nothing block with infinite loop */
  } while( true );
}

undefined8 FUN_0010080a(void)

{
  int iVar1;
  size_t sVar2;
  long in_FS_OFFSET;
  int local_b0;
  int local_ac;
  undefined8 local_a8;
  undefined8 local_a0;
  undefined8 local_98;
  undefined8 local_90;
  undefined local_88;
  undefined8 local_78;
  undefined8 local_70;
  undefined8 local_68;
  undefined8 local_60;
  undefined local_58;
  char local_48 [40];
  long local_20;
  
  local_20 = *(long *)(in_FS_OFFSET + 0x28);
  local_a8 = 0x3930343965353738;
  local_a0 = 0x3861623131383966;
  local_98 = 0x3665656562303635;
  local_90 = 0x3264373763306266;
  local_88 = 0;
  local_78 = 0x5a53010106040309;
  local_70 = 0x5c585354500a5b00;
  local_68 = 0x555157570108520d;
  local_60 = 0x5707530453040752;
  local_58 = 0;
  local_b0 = 0;
  while( true ) {
    sVar2 = strlen((char *)&local_a8);
    if (sVar2 <= (ulong)(long)local_b0) break;
    *(byte *)((long)&local_78 + (long)local_b0) =
         *(byte *)((long)&local_78 + (long)local_b0) ^ *(byte *)((long)&local_a8 + (long)local_b0);
    local_b0 = local_b0 + 1;
  }
  puts("Enter the secret password:");
  fgets(local_48,0x21,stdin);
  iVar1 = strcmp(local_48,(char *)&local_78);
  if (iVar1 == 0) {
    printf("You found the flag!\n247CTF{%s}\n",&local_78);
  }
  local_ac = 0;
  while( true ) {
    sVar2 = strlen((char *)&local_a8);
    if (sVar2 <= (ulong)(long)local_ac) break;
    *(undefined *)((long)&local_78 + (long)local_ac) = 0;
    local_ac = local_ac + 1;
  }
  if (local_20 != *(long *)(in_FS_OFFSET + 0x28)) {
                    /* WARNING: Subroutine does not return */
    __stack_chk_fail();
  }
  return 0;
}

Function Breakdown (FUN_0010080a):

The function appears to involve password checking, potentially XORing an encoded password with user input, and printing a flag if the password is correct. It also includes some stack protection.

Variables:

• iVar1: Integer used to store the result of strcmp (comparison between user input and the XORed value).
• sVar2: Used for storing the result of strlen, which measures the length of strings.
• in_FS_OFFSET: A variable that is probably used for stack protection.
• local_b0 and local_ac: Loop counters.
• local_a8 to local_60: Likely hexadecimal values representing encoded or obfuscated data.
• local_48 [40]: A buffer to store the user input (password).
• local_20: A variable to help detect stack corruption for security.

Initialization and Setup:

local_20 = *(long *)(in_FS_OFFSET + 0x28);
local_a8 = 0x3930343965353738;
local_a0 = 0x3861623131383966;
local_98 = 0x3665656562303635;
local_90 = 0x3264373763306266;
local_88 = 0;
local_78 = 0x5a53010106040309;
local_70 = 0x5c585354500a5b00;
local_68 = 0x555157570108520d;
local_60 = 0x5707530453040752;
local_58 = 0;
local_b0 = 0;

• local_a8 to local_60: These variables contain predefined hexadecimal values that might represent encrypted or obfuscated data, possibly the password.
• local_b0 = 0: Initializes a loop counter that will be used in the first loop.

First While Loop - XOR Operation:

while( true ) {
    sVar2 = strlen((char *)&local_a8);
    if (sVar2 <= (ulong)(long)local_b0) break;
    *(byte *)((long)&local_78 + (long)local_b0) =
         *(byte *)((long)&local_78 + (long)local_b0) ^ *(byte *)((long)&local_a8 + (long)local_b0);
    local_b0 = local_b0 + 1;
}

• Purpose: This loop runs through the characters in local_a8 and local_78, performing a bitwise XOR operation between the two.
  • local_a8 and local_78 contain predefined hexadecimal values. By XORing them, the code appears to be modifying local_78, which likely contains an encoded password or key.
  • Key Concept: XOR encryption is often used in simple encryption schemes to obfuscate or encode data.
• Flow:
  • The loop continues until all characters in local_a8 have been processed.
  • Each byte in local_78 is XORed with the corresponding byte in local_a8.

User Input and Password Check:

puts("Enter the secret password:");
fgets(local_48,0x21,stdin);
iVar1 = strcmp(local_48,(char *)&local_78);

• puts prints a prompt asking for the password.
• fgets reads up to 32 characters (0x21 or 33 including the null terminator) from standard input (user's password input) and stores it in local_48.
• strcmp compares the user's input (stored in local_48) with the now-XORed value in local_78.
  • If strcmp returns 0, it means the user input matches the XORed value in local_78.

Flag Output:

if (iVar1 == 0) {
    printf("You found the flag!\\n247CTF{%s}\\n",&local_78);
}

• If the password is correct (i.e., strcmp returns 0), the program prints a success message with the flag format 247CTF{} and the decoded content of local_78.

Additionally I’ve renamed the variables and removed the stack protection stuff from the pseudocode, so we can focus only on the primary functionality of the main function.

int main(void)

{
  int cmpResult;
  size_t strLength;
  int xorIndex;
  undefined8 encryptedPart1;
  undefined8 encryptedPart2;
  undefined8 encryptedPart3;
  undefined8 encryptedPart4;
  undefined xorKeyPadding1;
  undefined8 xorKeyPart1;
  undefined8 xorKeyPart2;
  undefined8 xorKeyPart3;
  undefined8 xorKeyPart4;
  undefined xorKeyPadding2;
  char inputBuffer [40];

  encryptedPart1 = 0x3930343965353738;
  encryptedPart2 = 0x3861623131383966;
  encryptedPart3 = 0x3665656562303635;
  encryptedPart4 = 0x3264373763306266;
  xorKeyPadding1 = 0;
  xorKeyPart1 = 0x5a53010106040309;
  xorKeyPart2 = 0x5c585354500a5b00;
  xorKeyPart3 = 0x555157570108520d;
  xorKeyPart4 = 0x5707530453040752;
  xorKeyPadding2 = 0;
  xorIndex = 0;
  while( true ) {
    strLength = strlen((char *)&encryptedPart1);
    if (strLength <= (ulong)(long)xorIndex) break;
    *(byte *)((long)&xorKeyPart1 + (long)xorIndex) =
         *(byte *)((long)&xorKeyPart1 + (long)xorIndex) ^
         *(byte *)((long)&encryptedPart1 + (long)xorIndex);
    xorIndex = xorIndex + 1;
  }
  puts("Enter the secret password:");
  fgets(inputBuffer,0x21,stdin);
  cmpResult = strcmp(inputBuffer,(char *)&xorKeyPart1);
  if (cmpResult == 0) {
    printf("You found the flag!\n247CTF{%s}\n",&xorKeyPart1);
  }

Dynamic Analysis

Now that we have an idea of what the program is doing, let’s proceed with debugging to see what data is being pushed to the stack and how that data is being XOR’d so we can get the password. For this challenge I will be using Radare2, of course you are free to use which ever debugger you like.

Setup

Let’s setup Radare2 to run the binary in debug mode and have the program start at the main function by running the command r2 -A -c 'ood; db main; dc' encrypted_password. Next up (this is optional) I like to use Radare2 in Visual mode, let’s switch over using the command V! . My view may be different from yours, as I have changed the two side panels to show the registers and stack (this can be done by click in the panel, right-clicking and scrolling down to Register and repeating the process for the panel below to change to Stack.)

The Encrypted Password

Now that the debugger is ready to go, let’s get started.

Proceed to step through the main function, as you can see the ASCII representation of the encrypted hex data we saw back in Ghidra is being moved into the rax and rdx registers then pushed to the stack at the addresses rbp-0xa0, rbp-0x98, rbp-0x90, and rbp-0x88 with the first XOR padding being pushed to rbp-0x80 . We can calculate the address of the encrypted data by taking the value of the base pointer rbp (0x7ffecc7aaad0) and subtracting it from their respective offsets (var_90h, var_98h, var_88h, etc...) using a hex calculator on ex. 0x7ffecc7aaad0 - 98 = 7FFECC7AAA38. As far as [s], [s1], and [s2] these are referred to as symbolic variable names.

Since the Byte Order of this system is Little Endian, the Least Significant Byte is being pushed to the stack first. For example less take a look at the first hex value “0x3930343965353738” being pushed to the stack:

XOR Key

With this knowledge, we now have an understanding of how the XOR key is being pushed to the stack as seen below. Before I continue let's talk about what a Bitwise XOR Operation is first

What is Bitwise XOR Operator?

In programming, a bitwise XOR is a binary operation that takes two bit patterns of equal length and performs the logical exclusive OR operation on each pair of corresponding bits. The result in each position is 1 if only one of the bits is 1, but will be 0 if both are 0 or both are 1. In this we perform the comparison of two bits, being 1 if the two bits are different, and 0 if they are the same. - Wikipedia. Here is an example using the encrypted hex data and the first XOR key.

The result of XORing two hex values byte-by-byte can be understood by looking at the binary representation of the hexadecimal numbers and the rules of the XOR (exclusive OR) operation. Here’s a step-by-step breakdown:

Example Byte: 39 XOR 5a

1. Convert the hexadecimal 39 and 5a to binary:
   • 39 (hex) = 00111001 (binary)
   • 5a (hex) = 01011010 (binary)
2. Perform the XOR operation:

This XOR operation results in 63 in hexadecimal, which corresponds to the ASCII character 'c'.

Why is the result the way it is?

For each byte pair, Ghidra or any binary analysis tool applies the XOR operation bit-by-bit. The result is a byte where the bits are the result of XORing the corresponding bits from the two original bytes.

For instance:

• The first bit of 39 is 0, and the first bit of 5a is 0, so the first bit of the XOR result is 0.
• The second bit of 39 is 0, and the second bit of 5a is 1, so the second bit of the XOR result is 1.

This process is repeated for each byte pair.

Why certain XOR results correspond to ASCII characters:

The resulting XOR value corresponds to a specific character in the ASCII table. For example:

• 63 in hexadecimal corresponds to 'c' in the ASCII table.
• 35 corresponds to '5'.
• If the resulting byte isn’t in the printable ASCII range, we usually display it as a dot (.), though in this case, all are within the printable range.

Decrypting the Password

Following the XOR key being pushed to the stack, an unconditional jump is made to an code segment that decrypts the encrypted data that was pushed to the stack. Once the jump is made, we land right before strlen gets called.

After stepping over the strlen function, I stop at the comparison being made between rax and rbx to see what values are being compared.

We can see, the rax register contains the hex value 20 (32 in decimal) and rbx contains the value 0. Following the comparison, a conditional jump is made stating if the value in the rbx register is below the value in the rax register, then make the jump back to 0x564867a008af.

After the jump is made, we finally land at the code segment that starts decrypting the password. Let’s take a look at what’s going on in this code segment

Decryption process

1. mov eax, dword [var_a8h]:
   • Move the 32-bit value from [var_a8h] into the eax register. (I find out later on that the value stored at rbp-0xa8 is actually the strlen counter)
2. movzx edx, byte [rbp + rax - 0x70]:
   • Zero-extend a byte from [rbp + rax - 0x70] and store it in edx. This instruction is loading a byte from memory and zero-extending it to fit the 32-bit edx register. (This is the XOR key being moved into the edx register.)
3. mov eax, dword [var_a8h]:
   • Move the 32-bit value from [var_a8h] to eax again (likely in preparation for the next operation).
4. movzx eax, byte [rbp + rax - 0xa0]:
   • Zero-extend the byte from [rbp + rax - 0xa0] into eax. The memory address is being computed relative to rbp. (This is the encrypted data being moved into the eax register.)
5. xor edx, eax:
   • Perform a bitwise XOR operation between edx and eax. The result is stored in edx.
6. mov eax, dword [var_a8h]:
   • Move the 32-bit value from [var_a8h] into eax again.
7. mov byte [rbp + rax - 0x70], dl:
   • Store the byte from the dl register (lower 8 bits of rdx) into memory at [rbp + rax - 0x70]. (The result of the XOR decryption)
8. add dword [var_a8h], 1:
   • Increment the value at [var_a8h] by 1. (Increases the strlen counter by 1)

The process above repeats 32 times, by the end we can see the fully decrypted flag in the stack.

Let's Connect!