Hasher 2

Platform: CUSTOM | Category: Reversing | Type: Challenge | Difficulty: Medium | OS: N/A | **Author: XESXOR Techniques: computed_goto_jump_table_analysis, greedy_per_byte_oracle_search, hashcat_mask_attack_on_md5, in_place_xor_transform_state_machine, loop_detection_early_stop, per_character_md5_positions_recovery, unicorn_concrete_oracle

Summary

Walkthrough of the CUSTOM Reversing Hasher 2. Task: x86-64 PIE keygen crackme with a two-stage licence check — an MD5 gate over selected characters plus a nested threaded state-machine serial validator (obfuscated computed-goto control flow). Sol

Description

Licence key format: SUUPER-K3YLIC-4NDMAN-YCHAR5-AFTE3R-ANDMOR Pass licence key in args...

A 64-bit ELF keygen crackme. The licence key is passed as argv[1]. The binary prints VALID! for a correct key and NON VALID! otherwise. Goal: recover a serial that validates. This is the harder variant of "Hasher": in addition to an MD5 gate it adds an obfuscated nested state-machine (check_serial).

Analysis

Recon

$ file hasher2.elf
ELF 64-bit LSB pie executable, x86-64, dynamically linked, not stripped, GCC (Ubuntu 7.4.0)

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Not stripped, so nm gives the map directly:

• Functions: main, check_format, check_serial, valid, non_valid, debug, and an MD5 implementation (md5_init/md5_update/md5_final/md5_transform).
• Data: b64t (a base64-style alphabet), state, length, buffer, initstate2.

Strings reveal the alphabet and the key format:

./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz   <- b64t
VALID!
NON VALID!
Licence key format: SUUPER-K3YLIC-4NDMAN-YCHAR5-AFTE3R-ANDMOR

valid prints VALID! then exit(0); non_valid prints NON VALID! then exit(0). ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Stage 1 — format + MD5 gate (in main)

The key layout is six 6-char groups joined by 5 dashes → total length 0x29 (41):

XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX
      ^      ^      ^      ^      ^        dashes at i where i%7 == 6

check_format iterates all 41 positions:

• at i%7==6 the char must be '-' (0x2d);
• otherwise the char must be in the base64 alphabet (0x2e-0x39 = ./0-9, 0x41-0x5a = A-Z, 0x61-0x7a = a-z).

main then computes MD5 over exactly the 6 chars at positions i%7==5, i.e. the LAST char of each group: positions [5,12,19,26,33,40]. It feeds each of those 6 bytes to md5_update, calls md5_final, and compares the 16-byte digest to a hardcoded array built by a run of mov byte ptr [rbp-0x20+..], imm (signed imm8s). Reconstructing the target digest from those signed bytes: ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

 python read-only • copy allowed Copy
signed = [0x17,-0xe,0x28,-0x65,0x1c,-0x5,-0x14,0x5b,
          0x56,-0x79,0x1c,0x15,-0x8,0x4e,0x58,-0x79]
target = bytes(b & 0xff for b in signed).hex()
# -> 17f2289b1cfbec5b56871c15f84e5887

If the MD5 does not match, main calls non_valid and exits before ever reaching check_serial.

Cracking it: it's the MD5 of a 6-char string from a 64-char alphabet (64^6 ≈ 6.9e10) — a hashcat mask attack finds it instantly:

 bash read-only • copy allowed Copy
echo '17f2289b1cfbec5b56871c15f84e5887' > md5.hash
hashcat -a 3 -m 0 md5.hash '?1?1?1?1?1?1' \
  -1 './0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz' -O
# 17f2289b1cfbec5b56871c15f84e5887:FENU3T

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

So the group-final chars are fixed: char5='F', char12='E', char19='N', char26='U', char33='3', char40='T'.

Stage 2 — check_serial (the hard part)

check_serial is a set of nested threaded state machines with obfuscated control flow. Primary dispatch is a computed goto through a jump table:

 asm read-only • copy allowed Copy
lea rdx, [rax*4]
lea rax, [rip+...]        ; jump table @ 0x9ce8
mov eax, [rdx+rax]
movsxd rax, eax
add rax, <base>
jmp rax

plus several chained cmp eax, K ; je addr dispatchers. State lives in a bank of local ints: the top state [rbp-0x80] (init 0xc0), a byte index [rbp-0x7c], a step counter [rbp-0x78], and many sub-machine states down to [rbp-0x04]. The tail near 0x9b3b checks that every state var is 0 and loops; the single success path (call valid at 0x9a14) only fires when the whole chain resolves. Crucially, check_serial never calls non_valid — on failure it just returns (⇒ not valid). ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Each transform+compare block does, with i = [rbp-0x7c]:

c = input[i]
optionally c ^= input[i+k]   # k in {1..5}: 'lea rdx,[rax+k]; movzx eax,[rax]; xor ecx,eax'  (29 such cross-byte XORs)
c ^= CONST                   # 'xor ecx, CONST'
input[i] = c                 # 'mov byte ptr [rax], dl'  (in-place store of transformed byte)
cmp input[i], TARGET         # 'cmp al, TARGET' -> ==TARGET => good branch (state A), else bad branch (state B)

There are ~352 transform blocks and 321 byte-comparison sites; some blocks unconditionally set the next state or advance the index. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Key behavioral insight (the crux): when the byte at the current index is wrong, the machine enters an infinite XOR-toggle loop at that index and never advances the max-reached index; when the byte is correct, it advances to the next index. This makes an oracle-guided search efficient.

Solution

Why not angr

Full symbolic exploration of check_serial times out — the nested dispatch loops cause path explosion. full_init_state on main and call_state on check_serial (even with veritesting) both hang. Do not rely on angr here. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Concrete Unicorn oracle for check_serial

Map each PT_LOAD segment at BASE+vaddr (BASE=0x400000; PIE so file offset == vaddr, and all in-function refs are RIP-relative or via the self-relative jump table, so any base works). Put the candidate serial in a buffer, point RDI at it (with a few trailing zero bytes since blocks read input[i+5]), set a fake return address, and hook valid/non_valid/normal-return:

 python read-only • copy allowed Copy
from unicorn import *
from unicorn.x86_const import *
import struct
‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

BASE, BUF, STACK = 0x400000, 0x1000000, 0x2000000
VALID, NON_VALID = BASE+0xe18, BASE+0xe32   # entry addrs of valid / non_valid
CHECK_SERIAL     = BASE+0x????              # from nm/objdump
RET_MAGIC        = 0x1337000

def load_segments(uc, path):
    with open(path,'rb') as f: data=f.read()
    # parse ELF64 program headers, for each PT_LOAD:
    #   map_size = align_up(memsz), uc.mem_map(BASE+vaddr, map_size)
    #   uc.mem_write(BASE+vaddr, data[off:off+filesz])
    ...

def make_oracle(path):
    uc = Uc(UC_ARCH_X86, UC_MODE_64)
    load_segments(uc, path)
    uc.mem_map(BUF, 0x10000)
    uc.mem_map(STACK, 0x10000)
    result = {'v': None}
‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

    def hook_code(uc, addr, size, _):
        if addr == VALID:     result['v']=True;  uc.emu_stop()
        elif addr == NON_VALID: result['v']=False; uc.emu_stop()
        elif addr == RET_MAGIC: uc.emu_stop()      # plain return => not valid
    uc.hook_add(UC_HOOK_CODE, hook_code)

    def run(serial):
        result['v']=None
        buf = serial.encode()+b'\x00'*8
        uc.mem_write(BUF, buf)
        uc.reg_write(UC_X86_REG_RDI, BUF)
        sp = STACK+0x8000
        uc.mem_write(sp, struct.pack('<Q', RET_MAGIC))  # fake return addr
        uc.reg_write(UC_X86_REG_RSP, sp)
        try: uc.emu_start(CHECK_SERIAL, 0)
        except UcError: pass
        return result['v'] is True
    return run

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Loop-detector + greedy per-byte search

A fixed instruction cap fails: a wrong byte loops until the 2M-instruction limit. Instead, hook the cmp al, imm8 sites to record which byte index is compared, and early-stop when many comparisons pass without the max index advancing ("stuck"). Then greedily force each unknown position left-to-right — keep the char that pushes the max-reached index furthest:

 python read-only • copy allowed Copy
ALPHABET = './0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'

def make_probe(path):
    # like make_oracle, but the code hook also:
    #   - detects 'cmp al, imm8' opcodes at the current index -> reads [rbp-0x7c] as idx
    #   - tracks max_idx; counts comparisons since last advance
    #   - if count > ~400 with no advance -> uc.emu_stop() (stuck), return max_idx
    # probe(serial) -> (validated: bool, max_idx: int)
    ...
‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

def greedy(path):
    probe = make_probe(path)
    # fixed by earlier stages: dashes at i%7==6, group-final chars = FENU3T
    fixed = {5:'F',12:'E',19:'N',26:'U',33:'3',40:'T'}
    key = ['A']*41
    for i in range(41):
        if i%7==6: key[i]='-'
        elif i in fixed: key[i]=fixed[i]
    unknown = [i for i in range(41) if i%7!=6 and i not in fixed]  # 30 positions

    for pos in unknown:
        best_c, best_reach = None, -1
        for c in ALPHABET:
            key[pos]=c
            ok, reach = probe(''.join(key))
            if ok: return ''.join(key)
            if reach > best_reach:
                best_reach, best_c = reach, c
        key[pos]=best_c
    return ''.join(key)

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Each byte is uniquely forced, so the search converges directly to the valid serial. Assembled with the MD5 chars and dashes → the full 41-char key.

Verification

Unicorn oracle:

 python read-only • copy allowed Copy
assert make_oracle('hasher2.elf')('REDACTED') is True

Real binary (host macOS, target Linux x86-64) via Docker with linux/amd64 (Rosetta) emulation:

 bash read-only • copy allowed Copy
docker run --rm --platform linux/amd64 -v "$PWD":/w ubuntu:20.04 \
  /w/hasher2.elf 'REDACTED'
# -> VALID!

The serial spells readable leet: SPBCTF - YOU'RE - AMAZIN - HOW YOU - SOLVE3 - IT B34T. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

What did NOT work

• angr: both full_init_state on main and call_state on check_serial time out on the nested dispatch loops (path explosion). Veritesting doesn't help.
• Fixed-cap unicorn trace: a wrong byte loops until the 2M-instruction limit, so a plain instruction/event cap just times out. The loop-detection early-stop (declare "stuck" after N comparisons with no index advance) is what makes the oracle fast enough to drive a per-byte search. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Key Takeaways / Lessons

• Mastering computed_goto_jump_table_analysis, greedy_per_byte_oracle_search, hashcat_mask_attack_on_md5, in_place_xor_transform_state_machine, loop_detection_early_stop, per_character_md5_positions_recovery, unicorn_concrete_oracle was key to this challenge.
• Always check for common misconfigurations and verify assumptions.

---

Crafted by XESXOR Research • 2026-07-02 • CUSTOM • Hasher 2 • Difficulty: Medium