Run a Python Script Detached with Real-Time Output

Scrapers are rarely quick. A crawl of thousands of pages can run for hours, and if you started it in a terminal over SSH, closing your laptop kills the whole job. The fix is to run the Python script detached from your session so it keeps going in the background — while still capturing its output so you can watch progress in real time.

This guide covers the practical options, from a one-liner to a proper always-on service, plus the one setting that trips everyone up: unbuffered output.

The buffering gotcha (fix this first)

Detach a script and immediately tail its log, and you may see... nothing, for a long time. The script is working, but Python buffers its output when writing to a file or pipe instead of a terminal, flushing only in large chunks. Your "real-time" log arrives in bursts.

Two ways to force real-time output:

• Run Python with the -u flag (unbuffered): python3 -u script.py
• Or set the environment variable PYTHONUNBUFFERED=1

Inside your code you can also flush explicitly (print(msg, flush=True)) or, better, use the logging module, which is the right tool for long jobs anyway. Get this right before anything else, or every method below will look broken.

Option 1: nohup (the quick one-liner)

nohup ("no hangup") detaches a process from your terminal so it survives logout, and redirects its output to a file. It is the fastest way to background a scraper:

nohup python3 -u script.py > scraper.log 2>&1 &

Breaking that down:

• nohup — keep running after the terminal closes.
• -u — unbuffered output (see above).
• > scraper.log — send stdout to a log file.
• 2>&1 — send stderr to the same file, so errors are captured too.
• & — run in the background and return the prompt.

Watch it live with:

tail -f scraper.log

tail -f streams new lines as they're written. To stop the scraper, find it with ps aux | grep script.py and kill <pid> (the background job also prints its PID when it starts).

nohup is perfect for fire-and-forget runs. Its limit: you can't easily reconnect to the running process to interact with it — you only have the log.

Option 2: tmux or screen (detach and reattach)

For a job you might want to check on interactively — scroll back, see live console output, even drop into it — a terminal multiplexer is nicer. tmux is the modern choice (screen is the older equivalent and works the same in spirit).

tmux new -s scraper       # start a named session
python3 -u script.py      # run your scraper normally, in the foreground
# press Ctrl-b then d to DETACH — the script keeps running

Disconnect, close your laptop, come back tomorrow, then:

tmux attach -t scraper    # reattach and see it still running
tmux ls                   # list sessions if you forget the name

With screen the equivalents are screen -S scraper, Ctrl-a d to detach, and screen -r scraper to reattach. Multiplexers are ideal during development and for jobs you want to babysit, because you get the full live terminal back exactly as you left it.

Option 3: systemd (for scrapers that must always run)

For a scheduled or 24/7 scraper on a Linux server, don't rely on a terminal at all — make it a systemd service. systemd starts it on boot, restarts it if it crashes, and centralizes its logs. Create /etc/systemd/system/scraper.service:

[Unit]
Description=My scraping job
After=network-online.target

[Service]
Type=simple
User=scraper
WorkingDirectory=/opt/scraper
Environment=PYTHONUNBUFFERED=1
ExecStart=/opt/scraper/venv/bin/python /opt/scraper/script.py
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Then enable and manage it:

sudo systemctl daemon-reload
sudo systemctl enable --now scraper
sudo systemctl status scraper
journalctl -u scraper -f          # real-time logs, like tail -f

Note PYTHONUNBUFFERED=1 in the unit and journalctl -f for the live stream. This is the setup for anything that needs to survive reboots and self-heal after crashes. (For a scheduled run rather than a continuous one, pair a simpler service with a systemd timer or a cron job.)

Option 4: launch and stream from within Python

Sometimes you want a parent Python program to launch a scraper as a separate process and read its output as it comes. subprocess.Popen does this — here's a corrected, modern version of the classic snippet:

import subprocess

# fire-and-forget: detach and log to a file
with open("output.log", "w") as f:
    subprocess.Popen(["python3", "-u", "script.py"], stdout=f, stderr=subprocess.STDOUT)

Popen returns immediately (it doesn't wait), so the child runs in parallel with the parent. -u keeps the file fresh. To process the output line by line as it's produced instead of just logging it:

import subprocess

proc = subprocess.Popen(
    ["python3", "-u", "script.py"],
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    text=True,
)
for line in proc.stdout:          # streams in real time
    print("scraper:", line.rstrip())
proc.wait()

This pattern is handy when a controller script orchestrates several scrapers and reacts to their progress.

Option 5: process managers and containers

Beyond the built-ins, two approaches scale nicely:

• supervisor — a lightweight process manager (popular where systemd isn't available or inside containers) that keeps a script running, restarts it, and collects logs via a simple config file.
• Docker / cloud — package the scraper in a container with PYTHONUNBUFFERED=1 set, and let your orchestrator (Docker, Kubernetes, a cloud run service, or a scheduled task) handle lifecycle and log collection. Container logs stream to docker logs -f or your platform's log viewer.

Which should you use?

• One-off long run over SSH → nohup ... & with -u and tail -f.
• A job you want to check on live → tmux (detach/reattach).
• Always-on or scheduled, must self-restart → systemd (or a timer/cron).
• A parent program spawning scrapers → subprocess.Popen.
• Fleet of scrapers / containers → supervisor or your container platform.

Whatever you choose, set unbuffered output, write timestamped logs (the logging module beats print), and make sure errors land in the same log so a silent 3 a.m. crash isn't a mystery. If keeping a fleet of long-running scrapers healthy is more infrastructure than you want to own, scraping.pro operates them for you and delivers the results as a managed data service. For servers you do manage remotely, the companion guide on SSH from the Linux terminal covers connecting in the first place.