checksum

Platform: CUSTOM | Category: Reversing | Type: Challenge | Difficulty: Medium | OS: N/A | **Author: XESXOR Techniques: ld_preload_function_hooking, self_checksum_bypass_without_patching, fopen_fwrite_vfprintf_interception, dynamic_log_capture, docker_linux_amd64_execution

Summary

Walkthrough of the CUSTOM Reversing checksum. Task: heavily protected, self-checksumming x86-64 ELF whose 'important calculations' are logged to /dev/null and discarded; on-disk patching triggers a tamper trap. Solution: bypass the self-checksum

Description

This heavily protected binary does some important calculations. We need its log output!

We are given a single stripped 64-bit PIE ELF (checksum.elf). It performs "important calculations" but writes them to a log we cannot see. The goal is to recover that log output, which encodes the flag.

Analysis

file checksum.elf => ELF 64-bit LSB PIE executable, x86-64, dynamically linked, stripped (BuildID d1e60b81ff72b256f0ed725d14c2ffa9e3f73cd5).

Interesting strings and imports: ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

• Strings: /proc/self/exe, Error opening binary!, Binary has been tampered with!, This incident will be reported., ping -c 3 this.binary.was.tampered.with.spbctf.ru, /dev/null, Opening log... , Failed! Bailing out, Starting normal operation, please wait..., All done, deinitializing.
• Imports: system, syscall, fopen, fwrite, vfprintf, fileno, __fxstat, strncmp, __isoc99_sscanf.

Three protection layers stand out:

1. Self-checksum / anti-tamper. The binary opens itself via fopen("/proc/self/exe","rb") and checksums its own image. If the on-disk bytes were modified, it runs system("ping -c 3 this.binary.was.tampered.with.spbctf.ru"), prints "Binary has been tampered with! This incident will be reported." and bails. The flag (REDACTED) is a deliberate trap: patching the check on disk is exactly what breaks the self-checksum. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

2. Log routed to /dev/null. The "important calculations" are written via fopen("/dev/null","a"), so every per-iteration log line is silently discarded. We need to recover those discarded writes.

3. Anti-debug + VM obfuscation. The binary reads /proc/self/status and inspects TracerPid, so strace/ltrace yield nothing usable. The main work function (~0x7f2e) is a small stack-machine/VM (jump table in .rodata at 0xe13c) that builds the log strings byte-by-byte at runtime, which is why the log format strings are not present statically — this is the "heavily protected" part. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Solution

Key idea: do not patch the file. LD_PRELOAD injects code via the dynamic loader without changing a single byte on disk, so the /proc/self/exe self-checksum still passes. We hook the libc calls the program uses for logging (fopen, fwrite, vfprintf) and mirror every write that targets a non-stdout/ stderr stream (the /dev/null log) into a real file captured.log.

Hook library (hook.c)

 c read-only • copy allowed Copy
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <dlfcn.h>
#include <stdarg.h>
‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

static FILE* (*real_fopen)(const char*, const char*) = NULL;
static size_t (*real_fwrite)(const void*, size_t, size_t, FILE*) = NULL;
static int (*real_vfprintf)(FILE*, const char*, va_list) = NULL;
static FILE* logcap = NULL;

__attribute__((constructor))
static void init(void){
    real_fopen = dlsym(RTLD_NEXT, "fopen");
    real_fwrite = dlsym(RTLD_NEXT, "fwrite");
    real_vfprintf = dlsym(RTLD_NEXT, "vfprintf");
    logcap = real_fopen("/work/captured.log", "w");
}

FILE* fopen(const char* path, const char* mode){
    if(!real_fopen) real_fopen = dlsym(RTLD_NEXT, "fopen");
    fprintf(stderr, "[HOOK] fopen(\"%s\", \"%s\")\n", path?path:"(null)", mode?mode:"(null)");
    return real_fopen(path, mode);
}
‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

size_t fwrite(const void* ptr, size_t size, size_t nmemb, FILE* stream){
    if(!real_fwrite) real_fwrite = dlsym(RTLD_NEXT, "fwrite");
    if(logcap && stream != stdout && stream != stderr){
        real_fwrite(ptr, size, nmemb, logcap); fflush(logcap);
    }
    return real_fwrite(ptr, size, nmemb, stream);
}

int vfprintf(FILE* stream, const char* fmt, va_list ap){
    if(!real_vfprintf) real_vfprintf = dlsym(RTLD_NEXT, "vfprintf");
    if(logcap && stream != stdout && stream != stderr){
        va_list ap2; va_copy(ap2, ap);
        real_vfprintf(logcap, fmt, ap2); va_end(ap2); fflush(logcap);
    }
    return real_vfprintf(stream, fmt, ap);
}

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

The fopen hook also logs to stderr so we can confirm the program's file activity: fopen("/proc/self/exe","rb") (self-checksum), then Opening log... fopen("/dev/null","a"), then fopen("/proc/self/status","r") (anti-debug), after which normal operation completes.

Build and run (Linux x86-64)

The host was macOS, so we run inside a Linux amd64 container. debian:buster repos are archived (gcc would not install), so we use the gcc:8 image:

 bash read-only • copy allowed Copy
docker run --rm --platform linux/amd64 -v "$PWD":/work -w /work gcc:8 bash -c '
  gcc -shared -fPIC -o hook.so hook.c -ldl
  chmod +x checksum.elf
  LD_PRELOAD=/work/hook.so ./checksum.elf
  cat captured.log'

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Because nothing on disk changed, the self-checksum passes, the anti-debug check finds no tracer, and the full log is captured to captured.log.

Recovering the flag

captured.log contains two workloads:

• Workload 1 prints Doing work # N... done and Result: ... — decoy values.
• Workload 2 prints both Result: ... and The Target Result: N. The flag is the concatenation of the byte values from each The Target Result: N line.

 python read-only • copy allowed Copy
import re
flag = ''
for l in open('captured.log', 'rb').read().split(b'\n'):
    m = re.search(rb'The Target Result: (\d+)', l)
    if m:
        flag += chr(int(m.group(1)))
print(flag)   # spbctf{REDACTED}

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Key Takeaways / Lessons

• Mastering ld_preload_function_hooking, self_checksum_bypass_without_patching, fopen_fwrite_vfprintf_interception, dynamic_log_capture, docker_linux_amd64_execution was key to this challenge.
• Always check for common misconfigurations and verify assumptions.

---

Crafted by XESXOR Research • 2026-06-24 • CUSTOM • checksum • Difficulty: Medium