FlagBruter

Platform: CUSTOM | Category: Reversing | Type: Challenge | Difficulty: Medium | OS: N/A | **Author: XESXOR Techniques: keyspace_brute_force, rc4_ksa_prga_recognition, stack_stored_ciphertext, xor_stream_decryption

Summary

Walkthrough of the CUSTOM Reversing FlagBruter. Task: not-stripped C ELF that RC4-decrypts a 24-byte stack-stored ciphertext using numeric keys 0..99999. Solution: recognize RC4 KSA/PRGA in the mangled hooyeah/yahoo functions, reimplement RC4 in Py

Description

Language: C. Download binary: flagbruter.elf. Sometimes the string variable contains flag. Most of times not.

A 64-bit ELF that, for each numeric key from 0 to 99999, RC4-decrypts a fixed 24-byte ciphertext stored on the stack. For exactly one key the result is a readable string — the flag. We must find that key.

Binary URL: https://rev-kids20.forkbomb.ru/files/rev/re5/flagbruter.elf

Analysis

file reports: ELF 64-bit LSB x86-64, dynamically linked, not stripped, BuildID cee429980d5111295992ee38562643fa107aa9bf. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Interesting symbols (C++-mangled, so the source mixed C and C++ naming):

Globals: S at 0x601080 (256-byte state array), i at 0x601180, j at 0x601184. Imports: malloc, free, memcpy, sprintf, __stack_chk_fail.

This is textbook RC4: ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

• hooyeah(key, keylen) = RC4 KSA. Initializes S[i] = i for i in 0..255, then runs j = (j + S[i] + key[i % keylen]) & 0xff with a swap of S[i]/S[j].
• yahoo() = RC4 PRGA. Each call does i = (i+1) & 0xff, j = (j + S[i]) & 0xff, swaps, and returns S[(S[i] + S[j]) & 0xff] — one keystream byte per invocation.

main() logic:

1. A 24-byte ciphertext is loaded onto the stack at rbp-0x70 via three movabs + qword stores:

   • 0x0572bed6666cc0e8
   • 0xab0d24d107c4dbce
   • 0x0076be26067eea1e ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

   Little-endian byte stream: e8 c0 6c 66 d6 be 72 05 ce db c4 07 d1 24 0d ab 1e ea 7e 06 26 be 76 00

2. Outer loop counter from 0 to 0x1869f (= 99999), i.e. keys "0".."99999".

3. Each iteration: sprintf(buf, "%d", counter) (format string "%d" at 0x400a54), then hooyeah(buf, 6) — RC4 KSA with keylen fixed at 6 (the digit string plus its NUL terminator, the remaining bytes of the 6-byte buffer staying zero). Then it decrypts 0x17 = 23 bytes by XORing each ciphertext byte with one yahoo() keystream byte into a malloc'd buffer, NUL-terminates it, memcpys to rbp-0x50, and frees. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

The challenge hint — "Sometimes the string variable contains flag. Most of times not." — tells us exactly one of the 100000 keys yields a meaningful plaintext. The binary itself only inspects the last iteration's result, so we just reimplement the loop and filter for a printable output.

Solution

Reimplement RC4 in Python, brute-force keys 0..99999 with the key being the decimal string zero-padded to 6 bytes (matching the C sprintf + 6-byte key buffer), decrypt the 24-byte ciphertext (first 23 bytes), and keep only printable / flag-like results. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

 python read-only • copy allowed Copy
#!/usr/bin/env python3
import struct

vals = [0x0572bed6666cc0e8, 0xab0d24d107c4dbce, 0x0076be26067eea1e]
ct = b''.join(struct.pack('<Q', v) for v in vals)

def ksa(k):
    S = list(range(256)); j = 0
    for i in range(256):
        j = (j + S[i] + k[i % len(k)]) & 0xff
        S[i], S[j] = S[j], S[i]
    return S

def stream(S, n):
    i = j = 0; out = []
    for _ in range(n):
        i = (i + 1) & 0xff
        j = (j + S[i]) & 0xff
        S[i], S[j] = S[j], S[i]
        out.append(S[(S[i] + S[j]) & 0xff])
    return out
‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

for n in range(100000):
    kb = (str(n).encode() + b'\x00' * 6)[:6]   # digit string + NUL padding, 6 bytes
    S = ksa(kb)
    ks = stream(S, 23)
    pt = bytes(c ^ k for c, k in zip(ct, ks))
    if all(32 <= b < 127 for b in pt) and (b'flag' in pt.lower() or b'{' in pt):
        print(n, pt)   # 47284 b'REDACTED'

Winning key: 47284 → plaintext REDACTED. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Key Takeaways / Lessons

• Mastering keyspace_brute_force, rc4_ksa_prga_recognition, stack_stored_ciphertext, xor_stream_decryption was key to this challenge.
• Always check for common misconfigurations and verify assumptions.

---

Crafted by XESXOR Research • 2026-06-24 • CUSTOM • FlagBruter • Difficulty: Medium