Friday 30 September 2011

Useful utility: Wget

Wget is a handy, free, open source, multi-platform utility for fetching files from HTTP, HTTPS and FTP servers.

It supports wildcards and recursion into subfolders. It also supports the sending of username and password information where required.

It operates as a command-line utility and there are lots of optional parameters (which can be confusing to be quite honest).

In the past I have used it to invoke a function on a remote web server on a scheduled basis (if the function can be initiated using a HTTP GET or POST).

It's clever and powerful and it just works.

All the information (and links to download) are available here.

Here are some examples of usage:
# Download the title page of example.com to a file
# named "index.html".
wget http://www.example.com/
 
# Download the entire contents of example.com
wget -r -l 0 http://www.example.com/