map / coding / log-parsing

Coding · principle 4 of 6

Log parsing

Stream the input, keep only the state you need, and you can process logs far larger than memory.

01 · Why it’s true

Logs are text with structure: every line has fields in a known order or format. Parsing means splitting each line into fields, filtering the ones you care about, and aggregating — counting, summing, or finding top values.

Processing one line at a time keeps memory use flat no matter how large the file is, which is why streaming tools (and streaming code) scale to huge logs. Before trusting an answer, handle the lines that don’t match the format.

02 · Picture it

1Read alinestream;memory staysflat2Splitfieldsskip linesthat don’tparse3Filtere.g. status5xx4Counthash map:path → n5Top Nsort or heap$awk '$9 ~ /^5/ {print $7}' access.log | sort | uniq -c | sort -rn | head
1Read a linestream; memory stays flat2Split fieldsskip lines that don’t parse3Filtere.g. status 5xx4Counthash map: path → n5Top Nsort or heap$awk '$9 ~ /^5/ {print $7}' access.log | sort | uniq -c | sort -rn | head
Stream, split, filter, count, report. In the common log format, field 9 is the status and field 7 the path.

03 · In the interview

  • A very common SRE coding question: parse a log and report something.
  • Showing both a quick shell one-liner and a robust script.
  • Handling malformed lines, huge files, and time windows.

04 · Practice question

“Given a web server access log, find the top 5 paths that returned 5xx errors in the last hour.”

A strong answer covers

  1. Stream the file; for each line parse timestamp, path, and status, and skip lines that don’t parse (count them).
  2. Keep lines from the last hour with a status from 500 to 599.
  3. Count per path in a hash map, then take the top 5 with a heap or a sort.
  4. Mention normalizing paths with IDs, and that a quick version is awk + sort | uniq -c.