← Back to Writeups
CUSTOMMediumCrypto

Четырёхслойный шифр (Four-Layer Cipher)

XESXOR9/10/20265 min read
#crypto#custom#medium#3d-coordinate-to-character-map#multi-layer-cipher-analysis-(b#leetspeak-pattern-recognition-#layer-specific-encoding-rules-

Четырёхслойный шифр (Four-Layer Cipher)

Platform: hackerlab | Category: Crypto | Type: Challenge | Difficulty: Medium | OS: NA | Author: XESXOR | Date: 2026-01-02 | Status: Solved Techniques: 3D coordinate to character mapping, Multi-layer cipher analysis (Bifid/Trifid/Quadrifid family), Leetspeak pattern recognition in partial plaintext, Layer-specific encoding rules identification

Summary

File four-layer.txt: 123 223 313 233 333 213 421 331 122 233 311 333 223 222 122 333 113 133 231 132 413 233

Четырёхслойный шифр (Four-Layer Cipher) — hackerlab

Description

"Bifid, Trifid... Quadrifid? Формат флага: CODEBY{flag}"

File four-layer.txt:

123 223 313 233 333 213 421 331 122 233 311 333 223 222 122 333 113 133 231 132 413 233

Analysis

Cipher Family Recognition

The description hints at a cipher family:

  • Bifid — uses a 5×5 Polybius square (2D)
  • Trifid — uses a 3×3×3 cube (3D, 27 positions)
  • Quadrifid — extension to a 4×4×4 cube (4D, 64 positions)

Data Analysis

  • 22 three-digit numbers
  • Each digit in range 1-4
  • Format: d1 d2 d3 — coordinates in 3D space ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Key Insight: Layer 4 is Special

When analyzing standard Trifid-like decoding:

  • Layers 1-3 (d1 = 1, 2, 3): standard formula index = (d1-1)*9 + (d2-1)*3 + (d3-1) → A-Z + underscore
  • Layer 4 (d1 = 4): encodes DIGITS, not letters!

This creates a leetspeak message where digits replace letters.

Solution

Decoding Algorithm

python
#!/usr/bin/env python3
"""
Quadrifid cipher decoder for hackerlab CTF
Layer 1-3: Letters (A-Z, _) via 3x3x3 cube
Layer 4: Digits via formula d2 + 2
"""

ciphertext = "123 223 313 233 333 213 421 331 122 233 311 333 223 222 122 333 113 133 231 132 413 233"
coords = ciphertext.split()
‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

# 27-character alphabet for layers 1-3 (3x3x3 = 27 positions)
alphabet27 = "ABCDEFGHIJKLMNOPQRSTUVWXYZ_"

result = ""
for c in coords:
    d1, d2, d3 = int(c[0]), int(c[1]), int(c[2])
    
    if d1 == 4:
        # Layer 4 -> digit using formula: d2 + 2
        # 421 -> d2=2 -> 2+2=4
        # 413 -> d2=1 -> 1+2=3
        result += str(d2 + 2)
    else:
        # Layers 1-3 -> letter via 3x3x3 cube
        # index = (d1-1)*9 + (d2-1)*3 + (d3-1)
        idx = (d1-1) * 9 + (d2-1) * 3 + (d3-1)
        result += alphabet27[idx]
‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

print(f"Decoded: {result}")
print(f"Flag: CODEBY{{{result.lower()}}}")

Step-by-Step Decoding

Coordd1d2d3LayerCalculationResult
1231231-3(0)*9 + (1)*3 + (2) = 5F
2232231-3(1)*9 + (1)*3 + (2) = 14O
3133131-3(2)*9 + (0)*3 + (2) = 20U
2332331-3(1)*9 + (2)*3 + (2) = 17R
3333331-3(2)*9 + (2)*3 + (2) = 26_
2132131-3(1)*9 + (0)*3 + (2) = 11L
4214214d2 + 2 = 2 + 2 = 44
3313311-3(2)*9 + (2)*3 + (0) = 24Y
1221221-3(0)*9 + (1)*3 + (1) = 4E
2332331-317R
3113111-3(2)*9 + (0)*3 + (0) = 18S
3333331-326_
2232231-314O
2222221-3(1)*9 + (1)*3 + (1) = 13N
1221221-34E
3333331-326_
1131131-3(0)*9 + (0)*3 + (2) = 2C
1331331-3(0)*9 + (2)*3 + (2) = 8I
2312311-3(1)*9 + (2)*3 + (0) = 15P
1321321-3(0)*9 + (2)*3 + (1) = 7H
4134134d2 + 2 = 1 + 2 = 33
2332331-317R
‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Result: REDACTED

Leetspeak Interpretation

  • L4YERS = LAYERS (4 → A)
  • CIPH3R = CIPHER (3 → E)

Message: "FOUR LAYERS ONE CIPHER" — describes the cipher itself!

Lesson Learned

Critical Mistake

When getting partial result FOUR_L?REDACTED?R, I assumed that unknown characters (?) should be letters (A and E) to form readable words "LAYERS" and "CIPHER".

What I Should Have Done

Consider ALL possible character types for unknowns:

  • Letters (A-Z)
  • Digits (0-9) ← initially missed
  • Special characters ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

CTF Pattern

In CTF challenges, leetspeak substitutions are very common:

  • 4 = A
  • 3 = E
  • 1 = I
  • 0 = O

When there's partial plaintext with unknowns, always try digits as a possibility, especially in multi-layer/multi-dimensional ciphers where different layers may encode different character types.

Rule

Don't fixate on one interpretation too early. Test simple formulas first (like d2 + 2) before complex ones.

References

  • Bifid cipher
  • Trifid cipher
  • Polybius square ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍