protected

Platform: spbctf | Category: Reversing | Type: Challenge | Difficulty: Medium | OS: Linux | Author: XESXOR | Date: 2026-06-24 | Status: Solved Techniques: binary_patching_nop_jne, bytecode_vm_reversing, jump_table_dispatch_identification, lea_displacement_redirect, md5_compression_recognition, self_printing_keygen_trick

Summary

Task: not-stripped x86-64 C crackme whose flag check is strcmp(input, genflag()) where genflag is a threaded-bytecode VM driving an MD5-like compression. Solution: skip reversing the VM by patching main to NOP the jne and redirect the success printf to the internal genflag output buffer, then run the binary so it prints its own expected flag.

protected — spbctf

Description

Flag is the input that makes the app say "Correct"

A 64-bit Linux ELF crackme (protected.elf), dynamically linked, not stripped, written in C. It reads a string from stdin and prints Congrats! on the correct input or Wrong! otherwise. The goal is to recover that correct input (the flag).

Analysis

Recon

$ file protected.elf
protected.elf: ELF 64-bit LSB executable, x86-64, dynamically linked,
  not stripped, for GNU/Linux 2.6.32, BuildID 781c4db566894fc40970329ec6dd7161035a5387
‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

$ strings protected.elf | grep -i flag
Congrats! Your flag is: %s
Wrong!

$ nm protected.elf | egrep 'genflag|main'
0000000000400856 T main
0000000000400eb4 T genflag
00000000006030a0 D _1_genflag_$array      # VM bytecode
0000000000603588 D _1_genflag_$strings    # embedded string fragments

The presence of genflag, a $array of bytecode, and a $strings data blob is the giveaway: the flag is generated at runtime, not stored as a constant.

main logic (0x400856)

1. scanf("%s", &input)            ; input at [rbp-0x90]
2. genflag(&buf)                  ; buf at [rbp-0x50], filled with target string
3. strcmp(input, buf)
   0x4008ad: jne  0x4008ca        ; -> "Wrong!"
   0x4008af: lea  rax,[rbp-0x90]  ; arg = user input
             ... printf("Congrats! Your flag is: %s", rax)

‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

Key insight (keygen pattern): the check is just strcmp(input, genflag_output). There is no hidden secret beyond whatever genflag deterministically produces. The success message even prints back the user's input — so if we make genflag's buffer be printed instead, the program reveals its own answer.

genflag (0x400eb4) — a threaded bytecode VM

genflag is not straight-line code; it is a small threaded interpreter / stack machine:

• Opcodes are read from _1_genflag_$array (0x6030a0).
• Primary dispatch is a computed jump: mov rax,[8*rax+0x401ab8]; jmp rax (jump table at 0x401ab8).
• A secondary 12-entry table at 0x4022a8 dispatches "call sub-commands" (strcpy / sprintf / fillbuf / strlen over string slots in _1_genflag_$strings).
• The VM has an operand-stack pointer at [rbp-0x230] and an instruction/data pointer at [rbp-0x220]. Opcodes cover push/pop, integer arithmetic (add/mul/xor, cmp+setl), multi-width dereference, string ops, and calls into helper routines. ‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

The crypto core — an MD5-like compression

The heavy helper is fillbuf (0x400a8f), a textbook MD5 compression function:

State words are initialized from constants at 0x603590; the round driver uses function-pointer tables (0x6035a0), K-index tables (0x6035c0), and shift amounts (0x6035e0 / 0x6035e8). Messages are padded with 0x80 then zeros, with the bit length (len*8) appended — the classic MD5 padding. ‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

Net effect: genflag uses MD5 plus the embedded string fragments to assemble flag_is_<md5-hex> at runtime.

Solution

Reversing the entire VM bytecode is unnecessary. Since the flag is exactly the deterministic output of genflag, the fastest path is the self-printing trick: patch main so the binary prints its own generated answer.

Two single-instruction patches in main:

1. Always take the success path. At vaddr 0x4008ad the bytes are 75 1b (jne 0x4008ca). Overwrite with 90 90 (two NOPs).
2. Redirect the printf argument from the user input buffer [rbp-0x90] to the genflag output buffer [rbp-0x50]. At 0x4008af the instruction is lea rax,[rbp-0x90] (48 8d 85 70 ff ff ff); change the displacement -0x90 → -0x50. ‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

For this binary the text segment maps at 0x400000 with file offset 0, so file_offset = vaddr - 0x400000.

 python read-only • copy allowed Copy
#!/usr/bin/env python3
import struct
data = bytearray(open('protected.elf', 'rb').read())
def off(v): return v - 0x400000

o = off(0x4008ad)                        # jne 0x4008ca -> nop nop
assert data[o:o+2] == bytes([0x75, 0x1b])
data[o] = 0x90; data[o+1] = 0x90

o = off(0x4008af)                        # lea rax,[rbp-0x90] -> [rbp-0x50]
assert data[o:o+3] == bytes([0x48, 0x8d, 0x85])
assert struct.unpack('<i', data[o+3:o+7])[0] == -0x90
data[o+3:o+7] = struct.pack('<i', -0x50)
‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

open('patched.elf', 'wb').write(data)
print('patched')

Running the Linux binary on macOS

The host is macOS; the binary is Linux x86-64. Run it via Docker with linux/amd64 (Rosetta) emulation:

 bash read-only • copy allowed Copy
echo "" | docker run --rm -i --platform linux/amd64 \
    -v "$PWD":/work ubuntu:20.04 /work/patched.elf
# -> Congrats! Your flag is: REDACTED

Verification

Feed the recovered string into the unmodified binary:

 bash read-only • copy allowed Copy
printf 'REDACTED' | \
  docker run --rm -i --platform linux/amd64 -v "$PWD":/work ubuntu:20.04 /work/protected.elf
# -> Congrats! Your flag is: REDACTED

‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

Confirmed correct.

What did NOT work

• gdb dynamic instrumentation in Docker on macOS failed. ptrace returns Couldn't get registers: Input/output error even with --cap-add=SYS_PTRACE --security-opt seccomp=unconfined, because Rosetta emulation under Docker Desktop does not support ptrace properly. The static patch + plain execution sidesteps debugging entirely. ‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍