DHIVA LABS · LEARN

Appearance

Lab: log detective ​

A web app had a bad morning, and you have its logs. Find out what went wrong with grep and pipes, watch a log live, catch a program holding a port and stop it, then solve the same case again with SysVitals and LogLens.

Level: beginner · Time: 45 to 60 minutes · Before this: the six lessons of this track

What you need ​

  • A Linux computer or virtual machine, with two terminal windows or tabs open. Everything here runs as your normal user. You never need sudo.
  • Python 3, which almost every Linux distribution includes. Check with python3 --version.
  • For part 5, SysVitals downloaded and executable. For part 6, LogLens installed from source, with its virtual environment. Parts 1 to 4 need neither.

Everything happens in one practice folder, ~/detective, which you delete at the end.

Part 1: set up the scene ​

  1. Make the practice folder and go into it:

    bash
    mkdir ~/detective
cd ~/detective

    The prompt now ends in ~/detective$.

  2. Create the app's log. Paste this whole block and press Enter:

    bash
    for i in $(seq 0 47); do
  t=$(printf '10:%02d:00' "$i")
  if   [ $((i % 10)) -eq 7 ]; then echo "2026-11-09 $t ERROR database connection refused"
  elif [ $((i % 15)) -eq 4 ]; then echo "2026-11-09 $t ERROR payment service timeout"
  elif [ $((i % 6)) -eq 0 ];  then echo "2026-11-09 $t WARN  slow response (1200 ms)"
  else                             echo "2026-11-09 $t INFO  request ok"
  fi
done > app.log

    You do not need to follow every detail. It is a loop: it counts i from 0 to 47, writes one log line per minute from 10:00 to 10:47, and picks the level from the remainder (%) of i divided by 10, 15 or 6. The > app.log at the end sends all 48 lines into the file. Nothing is printed.

  3. Create the web server's access log the same way:

    bash
    for i in $(seq 0 47); do
  case $((i % 12)) in
    5) code=500 ;;
    3|9) code=404 ;;
    *) code=200 ;;
  esac
  printf '192.168.1.%d - - [09/Nov/2026:10:%02d:00 +0000] "GET /page/%d HTTP/1.1" %d 512 "-" "curl/8.5.0"\n' $((10 + i % 7)) "$i" "$i" "$code"
done > access.log

    Each line is one web request in the common "combined" format: the client's address, the time, the request, the status code, the size, and the client program.

  4. Check what you made:

    text
    sam@laptop:~/detective$ wc -l *.log
  48 access.log
  48 app.log
  96 total
sam@laptop:~/detective$ head -n 3 app.log
2026-11-09 10:00:00 WARN  slow response (1200 ms)
2026-11-09 10:01:00 INFO  request ok
2026-11-09 10:02:00 INFO  request ok
sam@laptop:~/detective$ head -n 3 access.log
192.168.1.10 - - [09/Nov/2026:10:00:00 +0000] "GET /page/0 HTTP/1.1" 200 512 "-" "curl/8.5.0"
192.168.1.11 - - [09/Nov/2026:10:01:00 +0000] "GET /page/1 HTTP/1.1" 200 512 "-" "curl/8.5.0"
192.168.1.12 - - [09/Nov/2026:10:02:00 +0000] "GET /page/2 HTTP/1.1" 200 512 "-" "curl/8.5.0"

Part 2: question the logs ​

Answer each question with one command line. Try to build it yourself before you look at the answer under it.

  1. How many errors did the app log?

    text
    sam@laptop:~/detective$ grep -c ERROR app.log
8

  2. When did the trouble start? -m 1 makes grep stop after the first match:

    text
    sam@laptop:~/detective$ grep -m 1 ERROR app.log
2026-11-09 10:04:00 ERROR payment service timeout

  3. Which error happens most? Keep the error lines, cut off the date, time and level, then count the repeats:

    text
    sam@laptop:~/detective$ grep ERROR app.log | cut -d' ' -f4- | sort | uniq -c | sort -rn
      5 database connection refused
      3 payment service timeout

    The database is the main suspect.

  4. How healthy is the log overall? Count each level, the third word on every line:

    text
    sam@laptop:~/detective$ awk '{print $3}' app.log | sort | uniq -c
      8 ERROR
     32 INFO
      8 WARN

  5. What did the web server answer? In access.log the status code is the ninth word:

    text
    sam@laptop:~/detective$ awk '{print $9}' access.log | sort | uniq -c | sort -rn
     36 200
      8 404
      4 500

    Four requests failed on the server side (500), and eight asked for pages that do not exist (404).

  6. Which client got the most 404s? awk can filter too: $9 == 404 keeps only those lines, and {print $1} prints the client's address:

    text
    sam@laptop:~/detective$ awk '$9 == 404 {print $1}' access.log | sort | uniq -c | sort -rn | head -n 1
      2 192.168.1.13

  7. Write up the evidence. Put the app's errors and the server's 5xx lines in one report:

    text
    sam@laptop:~/detective$ grep ERROR app.log > report.txt
sam@laptop:~/detective$ awk '$9 >= 500' access.log >> report.txt
sam@laptop:~/detective$ wc -l report.txt
12 report.txt

    The first command uses > to start the file; the second uses >> to add to it. With > twice, you would keep only the last four lines.

Part 3: watch it live ​

  1. In the first terminal, follow the app log and show only errors:

    bash
    tail -f app.log | grep ERROR

    tail -f starts with the last 10 lines, so you see one old error, then nothing more. The command keeps running and waits:

    text
    2026-11-09 10:47:00 ERROR database connection refused

  2. In the second terminal, go to the folder and play the app: write one error and one normal line.

    bash
    cd ~/detective
echo "2026-11-09 10:48:00 ERROR disk full" >> app.log
echo "2026-11-09 10:48:30 INFO  request ok" >> app.log

    In the first terminal, only 2026-11-09 10:48:00 ERROR disk full appears. The INFO line was written to the file too, but grep filtered it out.

  3. In the first terminal, press Ctrl + C to stop following. You get your prompt back, and wc -l app.log now says 50 app.log.

Part 4: the program on port 8099 ​

Now a mystery program is holding a port. You will play both parts: start it, then track it down.

  1. In the first terminal, start a small web server in the background, with its output going to a log of its own:

    text
    sam@laptop:~/detective$ python3 -u -m http.server 8099 --bind 127.0.0.1 > server.log 2>&1 &
[1] 4410

    [1] is the job number and 4410 the PID; yours will differ. -u makes Python write each line straight away instead of saving them up, and > server.log 2>&1 sends both output streams to server.log. --bind 127.0.0.1 keeps the server reachable from this computer only.

  2. Find out what is listening on port 8099:

    text
    sam@laptop:~/detective$ ss -tlnp | grep 8099
LISTEN 0      5          127.0.0.1:8099      0.0.0.0:*    users:(("python3",pid=4410,fd=3))

    The program is python3, its PID is 4410, and it listens on 127.0.0.1 only.

  3. Visit it. Open http://127.0.0.1:8099/ in a web browser on the same computer: you see a list of the files in ~/detective. If you have curl, curl -s http://127.0.0.1:8099/ | head -n 3 does the same in the terminal. Then read the server's log:

    text
    sam@laptop:~/detective$ tail -n 2 server.log
Serving HTTP on 127.0.0.1 port 8099 (http://127.0.0.1:8099/) ...
127.0.0.1 - - [09/Nov/2026 17:46:35] "GET / HTTP/1.1" 200 -

  4. Stop it politely with SIGTERM, using the PID you found (not 4410, unless that is yours):

    bash
    kill 4410

    Press Enter once more and the shell reports [1]+ Terminated with the command.

  5. Check that the port is free:

    text
    sam@laptop:~/detective$ ss -tlnp | grep 8099
sam@laptop:~/detective$

    No output means nothing is listening on 8099 any more.

Part 5: the same case in SysVitals ​

  1. Start the server again. Press ↑ until you reach the python3 -u -m http.server … line and press Enter. Note the new PID.

  2. In the second terminal, go to the folder that holds sysvitals.py and start it with ./sysvitals.py. You start on the Processes tab.

  3. Press 3 for the Ports tab. Press /, type 8099 and press Enter.

    You see one tcp row: address 127.0.0.1, port 8099, your user, and your PID with python3 under PID/PROGRAM. It is not yellow, because it listens on the loopback address only.

  4. Press 1 for Processes. Press /, type the PID and press Enter. The table narrows to the python3 process with that PID.

  5. Press x. The bottom line asks Signal PID … (python3)? with [t] SIGTERM [k] SIGKILL [Esc] cancel. Press t.

    The bottom line shows sent SIGTERM to and the PID: the same signal that kill sent in part 4.

  6. Press q to quit SysVitals. In the first terminal, press Enter: the job is reported as Terminated.

TIP

./sysvitals.py --dump prints all four views once as text, so you can search it with the tools from part 2. While the server runs, ./sysvitals.py --dump | grep 8099 prints its line from the --- Listening ports --- section, ending in the PID and python3.

See Network connections and ports and Watch and stop processes for more.

Part 6: the same case in LogLens ​

  1. In the second terminal, go to your cloned loglens folder and activate its virtual environment with source .venv/bin/activate, as in Install.

  2. Start a third log that keeps growing, five lines a second, using the sample script that comes with LogLens:

    bash
    python3 scripts/genlog.py ~/detective/live.log --format generic --rate 5 &

    genlog.py empties the file you name before it writes, so always give it a new name such as live.log, never app.log.

  3. Open all three logs in one view:

    bash
    loglens ~/detective/app.log ~/detective/access.log ~/detective/live.log

    The status bar at the top shows one coloured label per file. LogLens loads the last 50 lines of each file at the start, which is all of app.log and access.log, then new live.log lines keep arriving.

  4. Press e for errors only. You see the ERROR lines from app.log, the 500 lines from access.log (LogLens counts a 5xx status as an error), and errors with their tracebacks from live.log. The status bar shows ERRORS. Press e again to see everything.

  5. Press /, type refused and press Enter. Only lines containing refused stay, from every file, like grep on all three at once. Press Esc to clear the filter.

  6. Press c to open the error clusters panel. Look for the group for database connection refused with [×5] and the group for payment service timeout with [×3]: the same counts your pipeline found in part 2. The live.log groups keep growing while you watch. Press Esc to close the panel.

  7. Press q to quit LogLens. Stop the sample script with kill %1 in the same terminal.

See Track down errors for the histogram and for exporting what you find.

Break it, try it ​

Each of these takes a minute or two and shows you a mistake on purpose, where it is safe.

Two servers, one port. Start the server from part 4 in the background, then start the same command again in the foreground, without the & and the redirection. The second one fails with OSError: [Errno 98] Address already in use. Find the first with ss -tlnp | grep 8099 and stop it with kill.

Open to the network. Start the server without --bind 127.0.0.1. ss -tlnp | grep 8099 now shows 0.0.0.0:8099, and in SysVitals's Ports tab the row is yellow: other machines on your network could reach the files in ~/detective. Try this only on a network you trust, and stop the server at once.

The overwrite trap. Make a copy and sort it into itself:

text
sam@laptop:~/detective$ cp app.log copy.log
sam@laptop:~/detective$ sort copy.log > copy.log
sam@laptop:~/detective$ wc -l copy.log
0 copy.log

The shell emptied copy.log for the > before sort started reading it. That is why you never redirect into the file you are reading.

Locked out. Take away every permission and try to read the report:

text
sam@laptop:~/detective$ chmod 000 report.txt
sam@laptop:~/detective$ cat report.txt
cat: report.txt: Permission denied
sam@laptop:~/detective$ chmod 644 report.txt

You own the file, so chmod gives the permissions back. No sudo needed.

Careful with pkill. pkill -f http.server stops every process whose full command line contains http.server, which can include processes you did not mean, such as someone else's server or a shell running a script with those words in it. Prefer ss to find the exact PID, and kill that.

Clean up ​

  1. Make sure nothing from the lab is still running. Both of these should print nothing:

    bash
    ss -tlnp | grep 8099
pgrep -af genlog

    If either prints a line, stop that PID with kill.

  2. Look at what you are about to delete, then delete it:

    bash
    cd ~
ls ~/detective
rm -r ~/detective

    ls should list only the lab files, such as access.log, app.log, live.log, report.txt and server.log. rm -r removes the folder for good.

  3. Keep SysVitals and the LogLens folder if you want to use them again. To remove them, see their install pages.

Check yourself ​

QuizLab: log detective

Question 1 of 6

Which command line lists the most common error message first?

Score: 0

What you learned ​

  • grep, cut, awk, sort, uniq -c and wc -l answer real questions about a log, one pipe at a time.
  • > starts a file and >> adds to it; redirecting into the file you are reading empties it.
  • tail -f | grep watches a log for the lines you care about.
  • ss -tlnp finds the program on a port, and kill sends it SIGTERM.
  • SysVitals shows the same port and process and stops it with the same signal; LogLens merges several logs and counts repeated errors for you.

Next steps ​