Nucleus

Platform: CUSTOM | Category: Reversing | Type: Challenge | Difficulty: Medium | OS: N/A | **Author: XESXOR Techniques: flag_from_xor_keys, pe_resource_parsing, recursive_binary_extraction, single_byte_xor_bruteforce

Summary

Walkthrough of the CUSTOM Reversing Nucleus. Task: PE32+ Windows executable with nested binaries in RCDATA resources, each XOR-encoded. Solution: Recursive extraction of 21 layers, brute-forcing single-byte XOR keys by looking for MZ header; fla

Description

This lab specimen is mysterious... it seems to be evolving. Get to the bottom of it for me, will you?

Files provided:

• nucleus.tar.gz containing nucleus21.exe (PE32+ executable, x86-64, Windows, 303,616 bytes)

The challenge involves a Windows PE executable that contains nested binaries within its resources, each XOR-encoded with a single-byte key. The flag is encoded across the XOR keys of all 21 layers.

Analysis

Initial Reconnaissance

Extracted nucleus21.exe from the tarball and performed initial analysis: ‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

 bash read-only • copy allowed Copy
file nucleus21.exe
# PE32+ executable (console) x86-64, for MS Windows

strings nucleus21.exe | grep -E "(exe|Resource|File)"
# 10.exe
# Failed to get source path
# Failed to open source for reading
# Failed to read file
# BeginUpdateResourceA
# EndUpdateResourceW
# CopyFileA
# CreateFileA
# UpdateResourceA

PDB path found: C:\Users\beds\source\repos\nucleus\x64\Release\nucleus.pdb

The binary uses Windows Resource API functions — it reads itself, copies itself, and updates resources in the copy. This suggests a self-replicating/nesting mechanism. ‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

PE Section Analysis

Parsed PE headers and found 6 sections. The .rsrc section was enormous:

The resource directory contained two entries:

• RCDATA (type 10, id 101): 0x46C00 bytes (289,792 bytes) — the bulk of the file
• Manifest (type 24): standard XML manifest

XOR Decoding Discovery

The RCDATA resource started with bytes 30 27 ed 7d ... — not a recognizable format. ‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

Brute-forced single-byte XOR keys (0x00–0xFF) looking for an MZ header (0x4D, 0x5A):

 python read-only • copy allowed Copy
for key in range(256):
    decoded = bytes([b ^ key for b in rcdata[:2]])
    if decoded == b'MZ':
        print(f"Found key: 0x{key:02x}")
        break

XOR key 0x7D (}) decoded the resource into a valid PE32+ executable — another nucleus binary!

Recursive Structure

Each nucleusN.exe contained nucleus(N-1).exe XOR-encoded in its RCDATA resource, with a unique single-byte XOR key per layer. This is a matryoshka-style nesting pattern. ‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

Solution

 python read-only • copy allowed Copy
#!/usr/bin/env python3
"""
Nucleus - TAMUctf
Recursive PE resource extraction with XOR decoding
"""
import struct

def parse_pe_rsrc(data):
    """Parse PE headers and extract RCDATA resource"""
    # DOS header
    if data[:2] != b'MZ':
        return None, None
    
    pe_offset = struct.unpack('<I', data[0x3C:0x40])[0]
    
    # PE signature check
    if data[pe_offset:pe_offset+4] != b'PE\x00\x00':
        return None, None
    
    # COFF header
    coff_offset = pe_offset + 4
    num_sections = struct.unpack('<H', data[coff_offset+2:coff_offset+4])[0]
    optional_header_size = struct.unpack('<H', data[coff_offset+16:coff_offset+18])[0]
    
    # Optional header - find resource directory RVA
    opt_offset = coff_offset + 20
    magic = struct.unpack('<H', data[opt_offset:opt_offset+2])[0]
    
    if magic == 0x20b:  # PE32+
        rsrc_rva = struct.unpack('<I', data[opt_offset+112:opt_offset+116])[0]
    else:  # PE32
        rsrc_rva = struct.unpack('<I', data[opt_offset+96:opt_offset+100])[0]
    
    # Section headers
    section_offset = opt_offset + optional_header_size
    rsrc_section = None
    
    for i in range(num_sections):
        sec_off = section_offset + i * 40
        name = data[sec_off:sec_off+8].rstrip(b'\x00')
        va = struct.unpack('<I', data[sec_off+12:sec_off+16])[0]
        raw_size = struct.unpack('<I', data[sec_off+16:sec_off+20])[0]
        raw_ptr = struct.unpack('<I', data[sec_off+20:sec_off+24])[0]
        
        if name == b'.rsrc':
            rsrc_section = (va, raw_size, raw_ptr)
            break
    
    if not rsrc_section:
        return None, None
    
    rsrc_va, rsrc_size, rsrc_ptr = rsrc_section
    
    # Parse resource directory to find RCDATA
    def parse_resource_dir(offset, level=0):
        num_named = struct.unpack('<H', data[offset+12:offset+14])[0]
        num_id = struct.unpack('<H', data[offset+14:offset+16])[0]
        
        for i in range(num_named + num_id):
            entry_off = offset + 16 + i * 8
            id_or_name = struct.unpack('<I', data[entry_off:entry_off+4])[0]
            offset_to_data = struct.unpack('<I', data[entry_off+4:entry_off+8])[0]
            
            if level == 0 and id_or_name == 10:  # RCDATA type
                # Follow to next level
                next_off = rsrc_ptr + (offset_to_data & 0x7FFFFFFF)
                return parse_resource_dir(next_off, level + 1)
            elif level == 1:
                next_off = rsrc_ptr + (offset_to_data & 0x7FFFFFFF)
                return parse_resource_dir(next_off, level + 1)
            elif level == 2:
                # Data entry
                data_entry_off = rsrc_ptr + offset_to_data
                data_rva = struct.unpack('<I', data[data_entry_off:data_entry_off+4])[0]
                data_size = struct.unpack('<I', data[data_entry_off+4:data_entry_off+8])[0]
                data_offset = rsrc_ptr + (data_rva - rsrc_va)
                return data[data_offset:data_offset+data_size]
        
        return None
    
    rcdata = parse_resource_dir(rsrc_ptr)
    return rcdata, rsrc_section
‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

def find_xor_key(data):
    """Brute-force single-byte XOR key to get MZ header"""
    for key in range(256):
        if (data[0] ^ key) == 0x4D and (data[1] ^ key) == 0x5A:
            return key
    return None

def xor_decode(data, key):
    """XOR decode data with single-byte key"""
    return bytes([b ^ key for b in data])

def extract_all_layers(initial_file):
    """Extract all nested binaries and collect XOR keys"""
    with open(initial_file, 'rb') as f:
        data = f.read()
    
    xor_keys = []
    layer = 21
    
    while True:
        rcdata, _ = parse_pe_rsrc(data)
        if rcdata is None or len(rcdata) < 2:
            break
        
        key = find_xor_key(rcdata)
        if key is None:
            break
        
        xor_keys.append(key)
        data = xor_decode(rcdata, key)
        
        # Save extracted layer
        with open(f'nucleus{layer-1}.exe', 'wb') as f:
            f.write(data)
        
        print(f"Layer {layer} -> {layer-1}: XOR key 0x{key:02x} = '{chr(key) if 32 <= key < 127 else '?'}'")
        layer -= 1
    
    return xor_keys
‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

# Extract all layers
keys = extract_all_layers('nucleus21.exe')

# Flag is XOR keys read from innermost to outermost
flag = ''.join(chr(k) for k in reversed(keys))
print(f"\nFlag: {flag}")

Extraction Results

All 21 layers extracted from nucleus21.exe down to nucleus0.exe (13,312 bytes, the innermost binary with no further RCDATA).

XOR keys for each layer (from innermost to outermost):

Reading from innermost outward: gigem{REDACTED}

Lessons Learned

1. PE Resource Parsing - Windows PE files can embed arbitrary data in RCDATA resources; parsing requires understanding the resource directory structure
2. Single-byte XOR Bruteforce - When data appears encrypted, try XOR with known plaintext (MZ header for executables)
3. Matryoshka Pattern - Nested binaries are a common CTF pattern; automate extraction with recursive scripts
4. Flag Encoding in Keys - The encryption keys themselves can encode the flag, not just the decrypted content
5. RCDATA Significance - The flag references "RCD4Ta" (RCDATA) — a hint about the technique used ‍​‌‌​​​​‌​‌‌​​‌‌​​‌‌​​‌‌​​‌‌​​‌​​​​‌‌​​​​​‌‌​​‌‌​​‌‌​​‌​‌​‌‌​​‌​‌‍

Key Takeaways / Lessons

• Mastering flag_from_xor_keys, pe_resource_parsing, recursive_binary_extraction, single_byte_xor_bruteforce was key to this challenge.
• Always check for common misconfigurations and verify assumptions.

---

Crafted by XESXOR Research • 2026-03-19 • CUSTOM • Nucleus • Difficulty: Medium