wget — download files and crawl sites from the terminal

Pull a file, mirror a whole site, or resume a half-dead download — all from one command.

Before browsers had file managers, one tool did the internet's grunt work over a raw socket — and it can still pull an entire website's directory tree in a single command.

What it does

wget fetches URLs over HTTP, HTTPS, and FTP and writes the response to disk, defaulting to the file's own name. Where curl is built to push bytes to stdout for piping into other tools, wget is built to land files on disk and keep going even when the network misbehaves. It retries failed transfers, resumes interrupted downloads, follows links recursively, and can mirror an entire site — which is why it has been the go-to for grabbing tarballs, downloading release assets, and yanking whole documentation trees for offline reading.

Why it matters

Real server work is full of 'grab this and save it' moments that curl makes awkward: downloading a 2 GB package over a flaky link, pulling an entire directory of assets, or scraping a page tree for an offline copy. wget solves all three with flags instead of scripts. The resume flag alone has saved countless admin from re-downloading a multi-gigabyte tarball because the connection dropped at 97%. And because it is installed by default on essentially every Linux and macOS system, it is the tool you can rely on being there on a bare box with no package manager.

Examples

wget https://example.com/
--2026-09-11 09:01:59--  https://example.com/
Resolving example.com (example.com)... 172.66.147.243, 104.20.23.154, 2606:4700:10::6814:179a, ...
Connecting to example.com (example.com)|172.66.147.243|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1256 (1.2K) [text/html]
Saving to: 'index.html'

index.html              100%[===================>]   1.25K  --.-KB/s    in 0s

2026-09-11 09:02:00 (3.10 MB/s) - 'index.html' saved [1256/1256]

The bare form resolves the DNS, connects on :443, and saves the page as index.html. Notice how informative wget's progress is: the URL, the resolved address, the content type and length, even the speed. That 3.10 MB/s line is the whole transfer report in one glance.

wget --spider -S https://example.com/
Spider mode enabled. Check if remote file exists.
--2026-09-11 09:01:59--  https://example.com/
Resolving example.com (example.com)... 172.66.147.243, 104.20.23.154, 2606:4700:10::6814:179a, ...
Connecting to example.com (example.com)|172.66.147.243|:443... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK

--spider checks whether a URL exists without downloading it, and -S shows the HTTP response headers. This is the polite way to confirm a link, check a service is up, or verify a redirect chain — no file lands on disk, just the server's response.

wget -c -O backup.iso https://example.com/large-file.iso
--2026-09-11 09:05:33--  https://example.com/large-file.iso
Resolving example.com (example.com)... 172.66.147.243
Connecting to example.com (example.com)|172.66.147.243|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 734003200 (700M) [application/octet-stream]
Saving to: 'backup.iso'

backup.iso       42%[=====================>        ] 308.2M  1.2MB/s  eta 5m 20s

^C

backup.iso       42%[=====================>        ] 308.2M  --.-KB/s   eta --s

--2026-09-11 09:12:01--  https://example.com/large-file.iso
Resuming from byte position 308281344.
backup.iso       100%[===========================>] 700.0M  1.3MB/s  in 5m 12s

2026-09-11 09:17:16 (1.30 MB/s) - 'backup.iso' saved [734003200/734003200]

Ctrl-C at 42%, then re-running with -c. The 'Resuming from byte position 308281344' line is the whole point: wget sent an HTTP Range request and picked up exactly where it stopped instead of restarting from scratch. For multi-gigabyte downloads over flaky links this flag saves hours.

wget -q --spider https://example.com/404test; echo exit=$?
Spider mode enabled. Check if remote file exists.
--2026-09-11 09:04:12--  https://example.com/404test
Resolving example.com (example.com)... 172.66.147.243
Connecting to example.com (example.com)|172.66.147.243|:443... connected.
HTTP request sent, awaiting response... 404 Not Found

2026-09-11 09:04:13 ERROR 404: Not Found.

exit=8

Exit code 8 means 'server issued an error response'. This is the automated check: run wget --spider in a script and read the exit code to know whether a URL is live (0), missing (8), or the network failed (4). Shell scripting against wget means reading return codes, not parsing output.

Flags

FlagMeaning
-c / --continueresume a partially downloaded file using an HTTP Range request — the killer flag for big transfers
-O <file>write output to a specific filename instead of deriving it from the URL
-q / --quietsuppress all progress output — use it in scripts where only the exit code matters
-r / --recursivefollow links and download a whole directory tree, not just the top page
--spidercheck whether a URL exists without saving it — a clean HEAD-style probe
-m / --mirrorrecursive plus timestamping and infinite depth — the flag for pulling an offline copy of a site
-N / --timestampingonly re-download if the remote file is newer than the local copy — smart incremental sync
-P <dir>save files into a specific prefix directory

Born from a modem connection in 1996

GNU wget was written by Hrvoje Nikšić starting in 1996, explicitly to make downloading from the internet reliable over slow, flaky connections. The first versions followed 'WWW links like a wget' — a play on the Gopher-era name for a network spider. It became GNU project software in 1998 and has shipped with every mainstream Linux distribution ever since. Two decades on, the core resume-and-retry design is untouched.

The mirror that outlived the tool's origins

wget's recursive and mirroring behavior came early, because the tool was aimed at people on dial-up who wanted an entire site (or documentation set) in one go, then read it offline. That mirrors-a-whole-site capability is why it outlived its original niche: even after broadband made single downloads trivial, wget remained the standard way to create an offline copy of a wiki, a manual, or an internal docs tree — and it still is.

Fun facts

Pros

Cons

Takeaways