A Simple Stack VM?

Platform: CUSTOM | Category: Reversing | Type: Challenge | Difficulty: Medium | OS: N/A | **Author: XESXOR Techniques: opcode_recovery, transform_inversion, vm_bytecode_analysis

Summary

Walkthrough of the CUSTOM Reversing A Simple Stack VM?. Task: a stripped Linux ELF implements a custom stack-based VM that validates a flag through embedded bytecode and a hidden target table. Solution: recover the opcode meanings from the interpreter, iso

Description

A simple stack VM?

English summary: we are given a stripped 64-bit Linux ELF and need to reverse its custom virtual machine to recover the correct CODEBY{...} flag.

Recon

The title hint is important: VM here means a custom virtual machine, not VirtualBox or VMware.

Initial triage showed that vm.elf is a stripped 64-bit Linux ELF PIE with NX and Full RELRO. Useful strings immediately suggested a stack-machine design:

• Full stack
• Stack empty
• STACK[%hu]: %hu
• It's that flag!! :)
• This is a false flag :(
• How about checking your flag? :)
• You need to spell the flag!
• input prompts for C, O, and D ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

The two main areas worth reversing were:

• the opcode dispatcher / interpreter around 0x11dd
• the main VM loop around 0x1aa0

The embedded VM bytecode lives in .rodata near 0x2020, and the table used during validation is stored in .data at 0x40e0.

VM Analysis

Following the interpreter showed that the program does not execute native logic for the flag check. Instead, it reads 16-bit words from the bytecode stream and dispatches them as VM opcodes.

The most useful step was to label opcodes by watching how each handler manipulated the VM state: stack top, instruction pointer, comparison flag, and a data-table index. Strings such as Full stack, Stack empty, and STACK[%hu]: %hu made the stack semantics obvious, while the success and failure strings identified output opcodes. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

The recovered opcode mapping used by the challenge program was:

• 0x1315 = PUSH imm16
• 0x1316 = ADD
• 0x1318 = XOR
• 0x1320 = POP
• 0x1321 = SUB (b - a)
• 0x1533 = compare low byte, set flag
• 0x1534 = conditional jump if true
• 0x1535 = conditional jump if false
• 0x2835 = push next word from .data:0x40e0
• 0x1400 = read char
• 0x1406 = print intro
• 0x40ab = print false message
• 0x40ac = print success message
• 0x1337 = stop VM

That mapping was enough to understand the full verification program in .rodata. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Extracting the Check

Once the bytecode was interpreted symbolically, the flag check reduced to a simple per-character loop.

For each input byte ch, the VM performs:

 text read-only • copy allowed Copy
((ch + 0x33) ^ 0xBB) - 0x13

Then it compares the low byte of the result with the next constant loaded by opcode 0x2835 from .data:0x40e0.

The embedded target table is:

 text read-only • copy allowed Copy
ba 26 b9 b0 bb 24 02 2a 14 08 05 b1 b0 16 0d b5 11 0f 16 1f 28 f8

The opcode meanings become clear from the structure of the bytecode itself: ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

1. READ_CHAR pushes the next user byte.
2. PUSH imm16 0x33, ADD matches ch + 0x33.
3. PUSH imm16 0xBB, XOR matches ^ 0xBB.
4. PUSH imm16 0x13, SUB matches subtracting 0x13.
5. PUSH_DATA, then the compare opcode, matches checking the transformed byte against the table.
6. Conditional jumps choose either the failure path (0x40ab) or continue the loop until the success path (0x40ac).

So even though the binary looks like a VM challenge, the core check is just a fixed reversible byte transform applied to each flag character. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Solving

To recover the original character, invert the transform:

 text read-only • copy allowed Copy
ch = (((target + 0x13) ^ 0xBB) - 0x33) & 0xFF

Applying that to every byte in the .data:0x40e0 table yields:

 text read-only • copy allowed Copy
CODEBY{REDACTED}

The helper script already present in the task directory, tasks/hackerlab/a_simple_stack_vm/solve_vm.py, implements exactly this inversion. A minimal equivalent solver is:

 python read-only • copy allowed Copy
#!/usr/bin/env python3

table = [
    0xBA, 0x26, 0xB9, 0xB0, 0xBB, 0x24, 0x02, 0x2A, 0x14, 0x08, 0x05,
    0xB1, 0xB0, 0x16, 0x0D, 0xB5, 0x11, 0x0F, 0x16, 0x1F, 0x28, 0xF8,
]
‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

def invert_byte(x: int) -> int:
    return (((x + 0x13) ^ 0xBB) - 0x33) & 0xFF

flag = "".join(chr(invert_byte(x)) for x in table)
print(flag)

Expected output:

 text read-only • copy allowed Copy
CODEBY{REDACTED}

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Key Takeaways / Lessons

• Mastering opcode_recovery, transform_inversion, vm_bytecode_analysis was key to this challenge.
• Always check for common misconfigurations and verify assumptions.

---

Crafted by XESXOR Research • 2026-04-05 • CUSTOM • A Simple Stack VM? • Difficulty: Medium