If you run scrapers on a remote box — a VPS, a cloud instance, or a dedicated server — SSH is how you get in. It gives you an encrypted shell on the remote machine straight from your terminal, so you can deploy code, start jobs, and read logs without ever touching a control panel. This quick guide covers the SSH command syntax, the key-based login you should be using instead of passwords, a config file that saves you typing, and the errors that trip people up.
The basic SSH command
The client is almost certainly already installed on Linux and macOS (and on Windows via WSL, Git Bash, or the built-in OpenSSH client). The syntax is simply the username and the host:
ssh user@host
For a concrete example — connecting as deploy to a server at lx567.example.com — you'd run:
ssh deploy@lx567.example.com
The first time, SSH shows the server's fingerprint and asks you to confirm; type yes and it's saved to ~/.ssh/known_hosts. Then you're either prompted for a password or logged straight in via a key (next section).
If the server listens on a non-standard port, pass it with -p:
ssh -p 2222 deploy@lx567.example.com
Use keys, not passwords
Password logins are fine for a first connection, but for anything you use regularly — and especially for a server exposed to the internet — key-based authentication is both more secure and more convenient. You generate a key pair, put the public half on the server, and from then on you log in with no password prompt and nothing brute-forceable.
1. Generate a key pair. Ed25519 is the modern default — small, fast, and strong:
ssh-keygen -t ed25519 -C "you@datahix.com"
Accept the default location (~/.ssh/id_ed25519) and set a passphrase when asked (it encrypts the private key on disk). This creates two files: id_ed25519 (private — never share it) and id_ed25519.pub (public — safe to copy around).
2. Copy the public key to the server. The easy way:
ssh-copy-id deploy@lx567.example.com
That appends your public key to the server's ~/.ssh/authorized_keys. If ssh-copy-id isn't available, paste the contents of id_ed25519.pub into that file manually.
3. Connect. Now ssh deploy@lx567.example.com logs you in using the key — no password. For hardened servers, disable password authentication entirely in /etc/ssh/sshd_config (PasswordAuthentication no) so only keys work.
Save typing with an SSH config file
Retyping long hostnames, users, ports, and key paths gets old fast. Define shortcuts once in ~/.ssh/config:
Host scraper
HostName lx567.example.com
User deploy
Port 2222
IdentityFile ~/.ssh/id_ed25519
ServerAliveInterval 60
Now the whole connection collapses to:
ssh scraper
ServerAliveInterval 60 is worth calling out for scraping work: it sends a keep-alive packet every 60 seconds so an idle SSH session doesn't get dropped by a firewall while you're watching a long job. (That said, don't rely on the SSH session to keep a scraper alive — detach it with tmux, nohup, or systemd, as covered in the guide on running scripts detached.)
Moving files: scp and sftp
SSH also carries file transfers, which is how you deploy scraper code and pull results back. Both tools ride the same connection and honor your config shortcuts.
# copy a local file up to the server
scp scraper.py scraper:/opt/scraper/
# pull a results file back down
scp scraper:/opt/scraper/output.csv .
# interactive session for browsing/transferring
sftp scraper
For keeping a whole directory in sync, rsync -avz -e ssh ./project/ scraper:/opt/scraper/ is more efficient because it only transfers what changed.
Common errors and fixes
Permission denied (publickey) — the server rejected your key. Check the key is in the server's ~/.ssh/authorized_keys, that you're using the matching private key (ssh -i ~/.ssh/id_ed25519 ... or an IdentityFile entry), and that permissions are strict: chmod 700 ~/.ssh and chmod 600 ~/.ssh/id_ed25519. SSH refuses keys that are readable by others.
WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! — the server's fingerprint no longer matches the one saved in known_hosts. This is expected after a reinstall or IP reassignment (and a red flag if it's unexpected). Remove the stale entry with ssh-keygen -R lx567.example.com and reconnect.
Connection refused — nothing is listening on that port. The SSH service may be down, or you have the wrong port. Confirm the daemon is running and the port matches.
Connection timed out — a network or firewall is dropping the traffic. Check the host/IP, any security-group or firewall rules, and that the server is reachable at all (ping, if ICMP is allowed).
Host key verification failed — same root cause as the changed-identity warning; clear the old known_hosts entry.
A quick security checklist
- Prefer keys over passwords; disable password auth on internet-facing servers.
- Protect private keys with a passphrase and keep permissions at
600. - Change the default port and/or use a firewall to cut brute-force noise.
- Consider
fail2banto auto-ban repeated failed logins. - Never commit private keys to a repo or paste them into a scraper's config.
Wrapping up
That's the core of working with remote servers over SSH: ssh user@host to connect, key-based login for security and convenience, an ~/.ssh/config to make it effortless, scp/sftp/rsync to move code and data, and a handful of predictable errors with quick fixes. With that in place you can manage a fleet of remote scraping servers comfortably from the terminal. If you'd rather skip the server administration altogether, scraping.pro runs the infrastructure and hands you the data as a managed service.