Using wget to download HTML website

Replace with the website you want to download files from.

wget -r -np -k

The above command will download all the files it can find in that web directory, i.e. (html files) This can be helpful if your trying to move a simple HTML site.

The -r option means recursive, the -k option converts the links to local links after it downloads the page.

wget multiple links with random access times

Create a file “list.txt” that contains all the URLs you want to download and launch the following command

for i in cat list.txt ; do wget ${i} && sleep $(( ( RANDOM % 120 ) +1 )) ; done

It’ll now run and after each link will wait a random amount of time up to 120 seconds before downloading the next link. Change the number as needed.

Download file from the web using curl

The following command basically does the same thing as wget.  This can come in handy since OS X and some linux distros do not ship with wget by default.

curl -O -L

The two options do the following

-O, –remote-name Write output to a file named as the remote file
-L, –location Follow redirects (H)

How to Install wget on OS X

The wget tool is an extremely useful command that allows you to download files from websites from the command line.  Before you begin you should make sure you have Xcode installed.  If you don’t, just download it from the App Store, its free.

To install wget on your mac you first need to download the source code from here.  Next you need to extract the tarball.  You can do this by double clicking the file in Finder or you can run the following command in the terminal

tar -zxvf ~/Downlaods/wget-1.14.tar.gz

Run the rest of these commands in the Terminal app.

Next we will cd into the directory.

cd ~/Downloads/wget-1.14


./configure --with-ssl=openssl

Make it


and finally install it.

sudo make install

And of course make sure it works.

wget --help

Congratulations, you have successfully compiled wget from source.