Happy New Year

Platform: CUSTOM | Category: Reversing | Type: Challenge | Difficulty: Medium | OS: N/A | **Author: XESXOR Techniques: algorithm_reversal, binary_patching, bug_exploitation, static_analysis

Summary

Walkthrough of the CUSTOM Reversing Happy New Year. Task: ELF binary with anti-debug protection that encrypts a flag using XOR with position-dependent key and ROL-3 bit rotation. Solution: Reversed the algorithm (ROR-3 then XOR with 0x67+i), discovered

Description

Someone has encrypted the Santa's message to the every child on Christmas Eve. Because of the age he forgot what he has written before and need someone help. Are you able to save the child's holiday?

Files: happy_newyear (ELF 64-bit binary)

Initial Analysis

 bash read-only • copy allowed Copy
$ file happy_newyear
happy_newyear: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked

$ nm happy_newyear | grep -E "anti_debug|check_flag|transform"
0000000000001169 T anti_debug
0000000000001289 T check_flag
00000000000011a9 T transform1
0000000000001209 T transform2

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Key findings:

• Binary is not stripped
• Contains anti-debug protection using ptrace
• Key functions: main, check_flag, transform1, transform2, anti_debug
• Encrypted flag stored at address 0x4020 (35 bytes)

Algorithm Reverse Engineering

check_flag function

• Requires input length == 35 (0x23)
• Applies transform1, then transform2
• Compares result with encrypted_flag

transform1 (XOR transform)

 c read-only • copy allowed Copy
for (int i = 0; i < len; i++) {
    buf[i] ^= (0x67 + i);
}

transform2 (Rotate transform)

 c read-only • copy allowed Copy
for (int i = 0; i < len; i++) {
    buf[i] = (buf[i] << 3) | (buf[i] >> 5);  // ROL 3 bits
}

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Encrypted flag data

00 d0 30 70 28 18 b0 31 70 00 08 1a 61 d1 80 08
41 09 42 f8 d0 70 f2 c2 f9 55 06 36 4d 15 2e 65
25 75 a7

Bug Discovery

Critical finding: The binary uses SAR (arithmetic shift right) instead of SHR (logical shift right):

 asm read-only • copy allowed Copy
sarb $0x5, %al    ; Arithmetic shift - sign extends!

This creates a bug:

• SHR: Always fills with 0s (unsigned)
• SAR: Fills with sign bit (signed) - for bytes >= 128, fills with 1s

Impact

For positions 25-34, the encrypted values are:

0x55, 0x06, 0x36, 0x4d, 0x15, 0x2e, 0x65, 0x25, 0x75, 0xa7

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

With SAR, only values 0xFD, 0xFE, 0xFF are achievable for high-bit inputs, making these positions impossible to match with the original algorithm.

Conclusion: The encrypted_flag was created using SHR, but the verification uses SAR. This is a challenge bug.

Solution

Decryption Script

 python read-only • copy allowed Copy
#!/usr/bin/env python3
"""
Happy New Year - Reverse challenge solver
Task ID: 398

Algorithm analysis:
1. transform1: XOR each byte with (0x67 + position)
2. transform2: ROL 3 bits: (byte << 3) | (byte >> 5)
‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

To decrypt:
1. ROR 3 bits (reverse of ROL)
2. XOR with (0x67 + position)
"""

# Encrypted flag from binary at address 0x4020
encrypted_flag = bytes([
    0x00, 0xd0, 0x30, 0x70, 0x28, 0x18, 0xb0, 0x31,
    0x70, 0x00, 0x08, 0x1a, 0x61, 0xd1, 0x80, 0x08,
    0x41, 0x09, 0x42, 0xf8, 0xd0, 0x70, 0xf2, 0xc2,
    0xf9, 0x55, 0x06, 0x36, 0x4d, 0x15, 0x2e, 0x65,
    0x25, 0x75, 0xa7
])

def ror(byte, n):
    """Rotate right by n bits"""
    return ((byte >> n) | (byte << (8 - n))) & 0xFF

def rol(byte, n):
    """Rotate left by n bits"""
    return ((byte << n) | (byte >> (8 - n))) & 0xFF
‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

# Decrypt
decrypted = bytearray(len(encrypted_flag))

for i in range(len(encrypted_flag)):
    # Step 1: Reverse transform2 (ROR 3 instead of ROL 3)
    byte = ror(encrypted_flag[i], 3)
    
    # Step 2: Reverse transform1 (XOR with same key)
    key = (0x67 + i) & 0xFF
    byte ^= key
    
    decrypted[i] = byte

print(f"Decrypted flag: {decrypted.decode()}")

# Verify by re-encrypting
def encrypt(plaintext):
    result = bytearray(len(plaintext))
    for i in range(len(plaintext)):
        byte = plaintext[i] ^ ((0x67 + i) & 0xFF)
        byte = rol(byte, 3)
        result[i] = byte
    return result
‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

re_encrypted = encrypt(decrypted)
print(f"Match: {re_encrypted == encrypted_flag}")

Binary Patching (for verification)

To verify the flag works with the binary, patch SAR to SHR:

 bash read-only • copy allowed Copy
# Find SAR instruction offsets: 0x139e and 0x144e
# SAR opcode: C0 F8 05 -> SHR opcode: C0 E8 05

# Patch using dd or hex editor
printf '\xe8' | dd of=happy_newyear_patched bs=1 seek=$((0x139e+1)) conv=notrunc
printf '\xe8' | dd of=happy_newyear_patched bs=1 seek=$((0x144e+1)) conv=notrunc

After patching:

 bash read-only • copy allowed Copy
$ ./happy_newyear_patched
Enter flag: grodno{REDACTED}
Correct,congratulations!

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Lessons Learned

1. SAR vs SHR matters: Arithmetic vs logical shift can break algorithms for high-bit values
2. Challenge bugs happen: Sometimes the intended solution doesn't work due to implementation errors
3. Binary patching: When verification is broken, patching can confirm the correct answer
4. Static analysis first: Understanding the algorithm before running avoids anti-debug issues ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Key Takeaways / Lessons

• Mastering algorithm_reversal, binary_patching, bug_exploitation, static_analysis was key to this challenge.
• Always check for common misconfigurations and verify assumptions.

---

Crafted by XESXOR Research • 2026-01-11 • CUSTOM • Happy New Year • Difficulty: Medium