networking · difficulty ◆◆

wget — download files and crawl sites from the terminal

Pull a file, mirror a whole site, or resume a half-dead download — all from one command.

Before browsers had file managers, one tool did the internet's grunt work over a raw socket — and it can still pull an entire website's directory tree in a single command.

AI-generated draft · reviewed and edited

2026-09-11 · 5 min read

$ wget

What it does

wget fetches URLs over HTTP, HTTPS, and FTP and writes the response to disk, defaulting to the file's own name. Where curl is built to push bytes to stdout for piping into other tools, wget is built to land files on disk and keep going even when the network misbehaves. It retries failed transfers, resumes interrupted downloads, follows links recursively, and can mirror an entire site — which is why it has been the go-to for grabbing tarballs, downloading release assets, and yanking whole documentation trees for offline reading.

Why it matters

Real server work is full of 'grab this and save it' moments that curl makes awkward: downloading a 2 GB package over a flaky link, pulling an entire directory of assets, or scraping a page tree for an offline copy. wget solves all three with flags instead of scripts. The resume flag alone has saved countless admin from re-downloading a multi-gigabyte tarball because the connection dropped at 97%. And because it ships as a standard package on Debian and Ubuntu it is usually already there on a bare Linux box; macOS is the exception — it ships curl, not wget, so there you install it yourself.

Example

$ wget https://example.com/

--2026-10-15 12:37:04--  https://example.com/
Resolving example.com (example.com)... 104.20.23.154, 172.66.147.243, 2606:4700:10::6814:179a, ...
Connecting to example.com (example.com)|104.20.23.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'index.html'
index.html              [<=>                 ]       0  --.-KB/s               
index.html              [ <=>                ]     577  --.-KB/s    in 0s      
2026-10-15 12:37:05 (56.3 MB/s) - 'index.html' saved [577]

The bare form resolves the DNS, connects on :443, and saves the page as index.html. Notice how informative wget's progress is: the URL, the resolved address, the content type, and the speed. example.com now answers with an unspecified (chunked) length, so there is no [fetched/total] figure at the end and the byte count tracks whatever the page currently contains — this run saved 577 bytes at 56.3 MB/s.

$ wget --spider -S https://example.com/

Spider mode enabled. Check if remote file exists.
--2026-10-15 12:23:40--  https://example.com/
Resolving example.com (example.com)... 172.66.147.243, 104.20.23.154, 2606:4700:10::6814:179a, ...
Connecting to example.com (example.com)|172.66.147.243|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Thu, 15 Oct 2026 10:23:40 GMT
  Content-Type: text/html; charset=utf-8
  Connection: keep-alive
  Server: cloudflare
  last-modified: Tue, 13 Oct 2026 22:05:48 GMT
  allow: GET, HEAD
  Accept-Ranges: bytes
  Age: 9
  cf-cache-status: HIT
  CF-RAY: a4ae1c150d40d87b-VIE
  alt-svc: h3=":443"; ma=86400
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

--spider checks whether a URL exists without downloading it, and -S prints the server's whole response header block. This is the polite way to confirm a link, check a service is up, or verify a redirect chain — no file lands on disk, just the server's response. Dates, Age, cf-cache-status and CF-RAY change with every request, so yours will not match line for line.

$ wget -c -O backup.bin https://proof.ovh.net/files/10Mb.dat

--2026-10-15 12:36:06--  https://proof.ovh.net/files/10Mb.dat
Resolving proof.ovh.net (proof.ovh.net)... 141.95.207.211, 2001:41d0:242:d300::
Connecting to proof.ovh.net (proof.ovh.net)|141.95.207.211|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 10485760 (10M), 9488912 (9.0M) remaining [application/octet-stream]
Saving to: 'backup.bin'
[...progress bar updates once a second; the bar shows a + for the already-have part...]
2026-10-15 12:36:07 (11.7 MB/s) - 'backup.bin' saved [10485760/10485760]

The same command was run twice: the first pass was interrupted with Ctrl-C after 996,848 bytes, the second pass resumed it. The two lines that prove the resume are '206 Partial Content' — the server accepted a Range request instead of sending the whole file — and 'Length: 10485760 (10M), 9488912 (9.0M) remaining', which is wget reporting how much it still has to fetch. There is no separate 'resuming' banner to look for. For multi-gigabyte downloads over flaky links this saves hours.

$ wget -q --spider https://example.com/404test; echo exit=$?

exit=8

With -q there is no output at all — no spider banner, no progress, no error text — so the exit code is the whole result: 0 means the URL is live, 8 means the server issued an error response (here a 404), 4 means the network failed. Shell scripting against wget means reading return codes, not parsing output.

Common flags

-c / --continue: resume a partially downloaded file using an HTTP Range request — the killer flag for big transfers
-O <file>: write output to a specific filename instead of deriving it from the URL
-q / --quiet: suppress all progress output — use it in scripts where only the exit code matters
-r / --recursive: follow links and download a whole directory tree, not just the top page
--spider: check whether a URL exists without saving it — a clean HEAD-style probe
-m / --mirror: recursive plus timestamping and infinite depth — the flag for pulling an offline copy of a site
-N / --timestamping: only re-download if the remote file is newer than the local copy — smart incremental sync
-P <dir>: save files into a specific prefix directory

History

Born from a modem connection in 1996

GNU wget was written by Hrvoje Nikšić, with the first public release — Geturl 1.0 — in January 1996, explicitly to make downloading from the internet reliable over slow, flaky connections. The name became Wget with version 1.4.0 in November 1996: the author had learned of an earlier Amiga program called GetURL, so his own name was shortened to 'WWW' plus 'get'. Wget 1.4.3, in February 1997, was the first release published as part of the GNU project, with the copyright assigned to the FSF. Two decades on, the core resume-and-retry design is untouched.

The mirror that outlived the tool's origins

wget's recursive and mirroring behavior came early, because the tool was aimed at people on dial-up who wanted an entire site (or documentation set) in one go, then read it offline. That mirrors-a-whole-site capability is why it outlived its original niche: even after broadband made single downloads trivial, wget remained the standard way to create an offline copy of a wiki, a manual, or an internal docs tree — and it still is.

Fun facts

Pros & cons

pros

+ Resume, retry, and mirror work out of the box — designed for flaky networks
+ Downloads land directly on disk with sensible default filenames, no shell redirection
+ Ships as a standard package on Debian and Ubuntu builds; on macOS you install it yourself (macOS ships curl, not wget)

cons

− Defaults to downloading to disk, so it is clumsy for piping bytes straight into another command
− The flag surface is huge and the -r / -m recursion flags need care to avoid pulling more than you meant
− Single-threaded — large parallel fan-outs are better served by aria2 or a download manager

Takeaways

1Add -c to every large download so a dropped connection resumes instead of restarting
2Read the exit code in scripts — 0 is success, 4 network failure, 8 HTTP error like 404
3Use --spider -S to check a URL or inspect response headers without saving anything
4Mirror a whole site with `wget -r -np -nH -P <dir> <url>` for an offline copy
5Pair -O with -c when you need a specific filename and resumability on the same file

Related commands

← all learning