Very Strong Protected

Platform: spbctf | Category: Reversing | Type: Challenge | Difficulty: Medium | OS: Linux | Author: XESXOR | Date: 2026-06-25 | Status: Solved Techniques: ptrace_traceme_anti_debug_bypass, jne_to_nop_patch, stack_vm_disassembly, nested_vm_reversing, custom_hash_identification, dynamic_execution_under_qemu_user, unlink_race_avoidance_via_bind_mount, syscall_tracing_with_qemu_strace

Summary

Task: statically linked stripped x86-64 ELF with ptrace anti-debug, multi-layer custom bytecode VMs, a custom zero-K MD5 hash, and a self-deleting output file. Solution: NOP the ptrace jne check, run under qemu-user, and bind-mount a capture file over /tmp/flag.txt so the program's unlink() fails (EBUSY), recovering the 40-byte flag content.

Very Strong Protected — spbctf

Description

vsp.elf — статически слинкованный, stripped x86-64 ELF. https://rev-kids20.forkbomb.ru/files/rev/re7/vsp.elf

English summary: A single static, stripped x86-64 Linux binary. It computes a flag string internally, writes it to /tmp/flag.txt, and immediately deletes the file. There are three protection layers: a ptrace anti-debug check, a multi-level custom bytecode VM hiding the real logic, and a self-deleting output file. The goal is to recover the flag content. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Analysis

Recon

• file vsp.elf → ELF 64-bit LSB, x86-64, statically linked, stripped. Not UPX.
• Interesting strings: "Go out, dirty debugger!", "/tmp/flag.txt", "aefg012!", "%02x", "w", "/proc/self/exe".

Anti-debug — main @ 0x4009ae

main calls ptrace(PTRACE_TRACEME) (wrapper fcn@0x443220). If the return value is nonzero it prints "Go out, dirty debugger!" and exits; otherwise it enters the VM interpreter fcn@0x400e6e (the real logic).

The trap: under qemu-user (and on the ARM analysis host) ptrace(TRACEME) returns -1 (ENOSYS), which the binary misreads as "being traced", so it always bails with the decoy message. This simultaneously defeats strace/gdb. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

The outer VM — interpreter @ 0x400e6e

A stack-based bytecode VM:

• IP = qword[rbp-0x290], initialized to bytecode @0x6ce340 (in .data).
• Operand stack pointer = qword[rbp-0x2a0]; locals frame = [rbp-0x180].
• Dispatch: op = byte[IP]; idx = op-8; if idx>0xf5 re-loop; else jmp table[idx*8 + 0x4a4388] (246-entry jump table).
• ~38 opcodes: PUSH_IMM, PUSH_LOCAL_ADDR, LOAD/STORE byte/i32/qword, ADD/SUB/MUL/XOR, CMP setl, relative-dword BRANCH/JMP, and a CALL opcode (op 0x43=67) for libc-style operations via a sub-dispatch table @0x4a4b38 (fwrite/fputs/fopen/strlen/printf/fclose and a nested VM). ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

A Python disassembler extracted the bytecode (bytecode.bin, full listing in disasm.txt). The real program spans offsets 0..1404 and ends in HALT (op 0x14).

Decoded outer program logic

1. Builds the 8-byte string arr = "aefg012!" via 8 strcpy CALLs (each copies one wide char from a UTF-16LE table @0x4a4360).
2. CALL 9 = strlen(arr) = 8.
3. CALL 10 = a nested VM (fcn 0x401f61, dispatch @0x401fc3, table @0x4a4bc0, opcode bias 11, bytecode @0x6ce940) = custom_hash("aefg012!", 8) → pointer to a 16-byte digest.
4. A 4×4 nested loop: CALL 11 = printf("%02x", digest[k]) builds the 32-hex-char digest string.
5. 8 unrolled XOR ops transform arr in place: arr[i] ^= KEY[i], KEY = [7,9,7,0,111,88,65,126]; "aefg012!" ^ KEY = "flag_is_".
6. CALL 12 fopen("/tmp/flag.txt","w"); CALL 14 fwrite(...); the 40-byte content is "flag_is_" + 32 hex chars; CALL 16 fclose.
7. unlink("/tmp/flag.txt") — the file is deleted right after being written (the "very strong protection"). ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

The nested hash — custom zero-K MD5

The nested VM (with further sub-VMs for the round functions F/G/H/I and a verified standard rotate-left-32 sub-VM @0x6ce1c0) implements an MD5-shaped hash:

• Standard IV (0x67452301 / 0xefcdab89 / 0x98badcfe / 0x10325476).
• Standard shift tables (7,12,17,22 / 5,9,14,20 / 4,11,16,23 / 6,10,15,21).
• Standard F/G/H/I selectors and rotate.

CRUCIAL difference: all 64 MD5 K (sine-table) constants are ZERO — the K-table pointers (@.bss) are zero and the constants are entirely absent from the binary. So it is NOT standard MD5: MD5("aefg012!") = ae29a780... is the WRONG answer (an initial mistake). The real custom digest is REDACTED. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Solution

Static hand-emulation of the K=0 MD5 variant (with permuted schedules and 5+ nested VMs, plus floating-point helper VMs for number formatting) is error-prone. The reliable path is dynamic execution of the binary.

Host is ARM macOS, so run an amd64 Linux container with qemu-user.

Step 1 — patch the anti-debug

The conditional jump after the ptrace check at vaddr 0x4009e6 is jne 0x4009f6 (bytes 75 0e). NOP it to 90 90 so execution always falls through to the VM. File offset = 0x9e6 (load base 0x400000). Save as vsp_nodbg.elf. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

 python read-only • copy allowed Copy
#!/usr/bin/env python3
data = bytearray(open("vsp.elf", "rb").read())
off = 0x9e6                       # vaddr 0x4009e6, base 0x400000
assert data[off:off+2] == b"\x75\x0e"   # jne 0x4009f6
data[off:off+2] = b"\x90\x90"     # nop; nop
open("vsp_nodbg.elf", "wb").write(data)

Step 2 — beat the self-deleting file with a bind mount

The program writes the flag then unlink()s it. Bind-mount a capture file over /tmp/flag.txt so the unlink() fails with EBUSY and the content survives: ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

 bash read-only • copy allowed Copy
docker run --rm --platform linux/amd64 --privileged \
  --security-opt seccomp=unconfined -v "$PWD":/work -w /work debian:stable-slim bash -c '
    apt-get update -qq && apt-get install -y -qq qemu-user-static
    chmod +x vsp_nodbg.elf
    touch /work/cap.txt; touch /tmp/flag.txt; mount --bind /work/cap.txt /tmp/flag.txt
    qemu-x86_64-static ./vsp_nodbg.elf
    cat /work/cap.txt
    umount /tmp/flag.txt'

Step 3 — confirm with syscall trace

qemu-x86_64-static -strace confirms:

open("/tmp/flag.txt", O_WRONLY|O_CREAT|O_TRUNC) = 9
write(9, buf, 40) = 40
unlink("/tmp/flag.txt") = 0

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Captured 40-byte content of cap.txt:

REDACTED

Wrap in the event format → spbctf{REDACTED}. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍