← Back to Writeups
CUSTOMMediumReversing

tracee

XESXOR9/2/20264 min read
#reversing#custom#medium#anti_disassembly_bypass#control_flow_analysis#pattern_byte_extraction#static_analysis#xor_decryption

tracee

Platform: CUSTOM | Category: Reversing | Type: Challenge | Difficulty: Medium | OS: N/A | **Author: XESXOR Techniques: anti_disassembly_bypass, control_flow_analysis, pattern_byte_extraction, static_analysis, xor_decryption

Summary

Walkthrough of the CUSTOM Reversing tracee. Task: stripped C x86-64 binary that forks into a Unix-domain-socket server/client pair; the child sends 18 bytes, the parent compares each input byte against server_byte XOR 0x36. Solution: recover th

Description

Flag is the input that makes the app say "Correct!". Note: the app needs a filesystem supporting Unix domain sockets (it won't run in a VM shared folder).

Given: tracee.elf — ELF 64-bit LSB x86-64, dynamically linked, stripped, built with GCC (Ubuntu 5.4.0). Download: https://rev-kids20.forkbomb.ru/files/rev/re5/tracee.elf

Goal: find the input string that produces the Correct! branch.

Analysis

The challenge name "tracee" hints at a parent/child IPC design. Relevant imports: socket, bind, listen, accept, connect, fork, read, write, fgets, malloc, strncpy, unlink, perror. The binary uses a Unix domain socket backed by a file literally named sckt (which explains the note about needing a real filesystem). ‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

Control flow

  1. main (0x400b47) calls fork():

    • cmp dword[pid], 0; jne <client> — if pid != 0 (parent) → client path; if pid == 0 (child) → falls through to the server function at 0x400a16.
  2. Child / server (0x400a16):

    • unlink("sckt"), then malloc(0x14) (20 bytes).
    • Calls the buffer-fill function at 0x400cc0 to populate the buffer with 18 secret bytes.
    • Creates an AF_UNIX (family=1) SOCK_STREAM socket, strncpys the path "sckt" into sockaddr_un, then bind / listen / accept.
    • write(client_fd, buf, 0x12) — sends exactly 18 bytes to the client. ‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍
  3. Parent / client (0x400b8c):

    • Sets up its own AF_UNIX socket with path "sckt", prints get your string!, then fgets(input, 0x100, stdin).
    • connects to the server socket.
    • Loops i = 0 .. 0x11 (18 iterations):
      • read(sock, &c, 1) — one byte from the server.
      • eax = c; xor eax, 0x36 (instruction at 0x400c5f).
      • Compares dl (= c ^ 0x36) against al = input[i] ([rbp + i - 0x110]).
      • Mismatch → prints string at 0x400e4a ("oh, no"), returns 3.
      • All 18 match → prints string at 0x400e51 ("OK!"), returns 1 = Correct. ‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

So the required input is simply: input[i] = server_byte[i] ^ 0x36.

Anti-disassembly: extracting the hidden buffer

The fill function at 0x400cc0 is obfuscated to defeat linear disassembly. It writes the buffer one byte at a time with mov byte ptr [rax], imm8; inc rax, and between writes it inserts always-resolved cmp r12,r14 / je / jne / jmp rel8 gadget chains that weave the real instructions out of linear order. objdump only shows the first mov byte ptr [rax], 0x50; the rest are hidden behind the eb c6 short jumps. ‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

The gadget signature between byte writes is: 4d 39 f4 74 03 75 01 eb c6 (cmp r12,r14 / je / jne / jmp).

Recovery approach: scan the .text bytes for the pattern c6 00 XX (mov byte[rax], imm8) immediately followed by 48 ff c0 (inc rax). This yields the 18 immediate bytes in order. A manual control-flow trace through the eb c6 jumps confirms the same ordering.

Solution

python
#!/usr/bin/env python3
# Recover the 18 secret bytes that the server writes, then XOR with 0x36.
data = open('tracee.elf', 'rb').read()
‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

# Find every `mov byte ptr [rax], imm8` (c6 00 XX) directly followed by
# `inc rax` (48 ff c0). The imm8 values, in file order, are the buffer bytes.
buf = bytes(
    data[i + 2]
    for i in range(len(data) - 5)
    if data[i] == 0xc6 and data[i + 1] == 0x00
    and data[i + 3] == 0x48 and data[i + 4] == 0xff and data[i + 5] == 0xc0
)

print("hidden buffer:", buf.hex())
print("flag:", ''.join(chr(b ^ 0x36) for b in buf))

Output:

hidden buffer: 505a5751 4d415e59 59465f53 696e6e6e 154b
flag: flag{REDACTED}

‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

Per-byte XOR with 0x36:

50^36='f' 5A^36='l' 57^36='a' 51^36='g' 4D^36='{'
41^36='w' 5E^36='h' 59^36='o' 59^36='o' 46^36='p'
5F^36='i' 53^36='e' 69^36='_' 6E^36='X' 6E^36='X'
6E^36='X' 15^36='#' 4B^36='}'

(Verified end-to-end by running the binary on a real filesystem and feeding the recovered string — the app prints Correct!.) ‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

Key Takeaways / Lessons

  • Mastering anti_disassembly_bypass, control_flow_analysis, pattern_byte_extraction, static_analysis, xor_decryption was key to this challenge.
  • Always check for common misconfigurations and verify assumptions.

Crafted by XESXOR Research • 2026-06-24 • CUSTOM • tracee • Difficulty: Medium