Let's Avoid Doing Math

Platform: DawgCTF SP26 | Category: Misc | Type: Challenge | Difficulty: Easy | OS: NA | Author: XESXOR | Date: 2026-04-10 | Status: Solved Techniques: confusion_matrix_reconstruction, metric_calculation, output_format_normalization, specification_parsing

Summary

Task: a validation log records known and detected threat-depth classes for 120 malware samples, and the flag is built from per-class accuracy, FPR and FNR values. Solution: reconstruct the confusion matrix from the log, then report (accuracy, FPR, FNR) for each class in ascending severity order — the trap is that 'for each classification' applies to accuracy too, so the flag has nine per-class values and no single overall accuracy.

Let's Avoid Doing Math — DawgCTF SP26

Description

Last weekend was a red team exercise against our malware attack vector depth analysis tool. The details of how it works are trade secrets, all we really need is a just verbose enough report on how accurate it was, and what our false positive and false negatives rates were for each classification in growing order of importance. Just to be clear, our boss is picky, they want exactly a single leading zero and no trailing zeros for these values. They also said something about all values needing to be comma separated and all. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

The challenge ships a single file, threat_depth_analysis.log, containing 120 labeled malware sample records. Each record has both the ground-truth Known Threat Depth and the tool's Detected Threat Depth, chosen from three classes: minor, medium, major. Our job is to compute the classification metrics and serialize them into the flag in the very specific format the "picky boss" wants.

Recon

The log is plain text, 120 sample blocks of the form:

--- Sample Analysis #001 ---
Sample ID: sample_001
SHA256: 00000001deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef00000011
Known Threat Depth: minor
Detected Threat Depth: minor
Confidence Score: 0.84
Behaviors Detected: suspicious API calls, registry modification
Analysis Timestamp: 2026-04-03 22:30:00 UTC
----------------------------------------

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Only the Known Threat Depth and Detected Threat Depth lines matter. Parsing them with two regex finds yields 120 paired labels, 40 samples per class.

Confusion matrix

Totals: 120 samples, 99 correct, 21 misclassified.

Per-class metrics

The three quantities requested by the boss, computed one-vs-rest:

Every value comes out to a clean terminating decimal, which lines up neatly with the "exactly a single leading zero and no trailing zeros" hint — str() / repr() in Python already produces the required form (0.05, 0.1, 0.1125, 0.85, …).

The actual challenge: parsing the sentence

The arithmetic really is trivial. The hard part — the whole point of the task — is parsing the English spec:

"a just verbose enough report on how accurate it was, and what our false positive and false negatives rates were for each classification in growing order of importance" ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

There are two plausible parses:

1. Naive (wrong): "one overall accuracy" + "FPR and FNR for each classification". This gives 1 + 3 + 3 = 7 values and leads you straight to DawgCTF{0.82REDACTED} (or some permutation of it). This is wrong.
2. Correct: "for each classification" scopes the entire list of metrics, including accuracy. That is: for each classification, report how accurate it was and what the FPR and FNR were. That gives 3 × 3 = 9 values and no single overall accuracy at all. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

The "just verbose enough" phrasing and the mention of "a report" reinforce parse #2 once you notice it: a proper per-class report has an accuracy per class, not just a global one. The distinguishing numbers — 0.95, 0.85, 0.85 — only appear under parse #2.

Once the parse is right, everything else follows the most obvious reading:

• Classes in ascending severity (minor → medium → major) — this is "growing order of importance".
• Within each class, output accuracy, FPR, FNR (the exact order the sentence lists the three metrics).
• Comma-separated, one leading zero, no trailing zeros. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Solution

 python read-only • copy allowed Copy
#!/usr/bin/env python3
"""Solver for DawgCTF SP26 — Let's Avoid Doing Math."""
import re
from collections import Counter
from pathlib import Path

text = Path("threat_depth_analysis.log").read_text()
known    = re.findall(r"Known Threat Depth:\s*(\w+)",    text)
detected = re.findall(r"Detected Threat Depth:\s*(\w+)", text)
assert len(known) == len(detected) == 120

classes = ["minor", "medium", "major"]  # growing order of importance
pairs = Counter(zip(known, detected))
N = len(known)
‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

values = []
for c in classes:
    TP = pairs[(c, c)]
    FN = sum(pairs[(c, o)] for o in classes if o != c)
    FP = sum(pairs[(o, c)] for o in classes if o != c)
    TN = N - TP - FP - FN
    acc = (TP + TN) / N          # per-class binary accuracy
    fpr = FP / (FP + TN)
    fnr = FN / (FN + TP)
    values += [acc, fpr, fnr]

# Python's default str() already yields "one leading zero, no trailing zeros"
# for every one of these clean fractions.
flag = "DawgCTF{" + ",".join(str(v) for v in values) + "}"
print(flag)

‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Running it prints:

 text read-only • copy allowed Copy
DawgCTF{REDACTED}

Pitfalls / wrong attempts

Every wrong attempt during the solve was a variant of the 7-value "one overall accuracy" parse. All four were rejected by the scoreboard:

1. DawgCTF{0.82REDACTED} — overall accuracy, per-class interleaved FP first, ascending importance.
2. DawgCTF{0.825,0.05,0.1,0.1125,0.05,0.25,0.225} — overall accuracy, grouped (all FPRs then all FNRs), ascending importance.
3. DawgCTF{0.82REDACTED05,0.05} — overall accuracy, descending importance interleaved FP first.
4. DawgCTF{0.825,0.1,0.05,0.2,0.25,0.225,0.225} — tried rescuing the 7-value format with a non-standard FP/40, FN/40 formula. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

None of these can be right, because once you commit to the 9-value per-class reading, the first natural ordering — classes ascending, tuple (accuracy, FPR, FNR) — is already correct and there is no room left to permute.

Lesson learned: when a challenge says "how accurate it was, and … for each classification", read "for each classification" as scoping the entire enumeration. Phrases like "a just verbose enough report" are a hint that the flag contains a proper per-class report, not a single global metric with a handful of helpers glued on. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍

Key indicators

Use this technique when:

• the challenge hands you a log of ground-truth vs predicted labels and asks for a metric summary,
• the flag must be assembled from a small set of numeric values with strict formatting,
• the description carefully lists multiple metrics followed by "for each classification / class / category" — the scope of that phrase is the whole puzzle. ‍​‌‌​​‌​​​‌‌​​‌​​​​‌‌​‌​​​​‌‌‌​​‌​​‌‌​​​‌​​‌‌‌​​​​​‌‌​‌‌​​​‌‌​​​​‍