Wget is a non-interactive network downloader which can be used for downloading files in Unix/Linux. It supports the HTTP, HTTPS and FTP protocols and also has proxy support.
The wget command is quite powerful and has a lot of options available, in this guide we’ll be covering 20 of the most important wget examples that will help you learn how to best use it.
Install Wget
It is not uncommon for wget to not be installed by default, if the command is not available for you then you must first install it as shown below.
CentOS/RHEL/Fedora
[root@centos7 ~]# yum install wget -y
Debian/Ubuntu
root@ubuntu:~# apt-get install wget -y
How To use wget – Command Examples
Now that we have wget installed and ready to use, here are our top wget command examples.
1. Download A File
Lets begin with a basic example of downloading a simple file from the Internet. We simply specify the file that we want to download after the wget command, as shown below.
[root@centos7 Downloads]# wget https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso --2016-08-31 11:30:54-- https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 394264576 (376M) [application/octet-stream] Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso’ 68% [=========================> ] 271,936,588 2.32MB/s eta 54s
The output provides us with percentage complete, download speed, and time remaining estimate.
This has downloaded a copy of the file to the current working directory with the same file name as was in the URL.
[root@centos7 Downloads]# ls CentOS-7-x86_64-NetInstall-1511.iso
2. Download To Specific Directory With New Name
We can use the -O option to specify the output file, allowing us to tell wget where to download the file to, and what to name the file. This saves us having to download it to the current working directory, move it, and then potentially rename it as may be required if following the first example.
[root@centos7 Downloads]# wget -O /root/file.txt https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt --2016-08-31 11:36:25-- https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 406 [text/plain] Saving to: ‘/root/file.txt’ 100%[==============================================================================>] 406 --.-K/s in 0s 2016-08-31 11:36:25 (139 MB/s) - ‘/root/file.txt’ saved [406/406]
Here we have downloaded the remote file md5sum.txt and saved it to /root/file.txt.
3. Limit Download Speed
Rather than letting wget download at maximum speed which is the default, we can optionally cap it at a specific speed with the --limit-rate option. This may be useful if you don’t want wget to use all available network bandwidth.
The speed limit can be defined in a number of bytes, or kilobytes with the k suffix, or megabytes with the m suffix.
In the below example, we start downloading the .ISO file with a 500KB/s limit, and this is confirmed in the wget output.
[root@centos7 Downloads]# wget --limit-rate=500k https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso --2016-08-31 11:37:47-- https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 394264576 (376M) [application/octet-stream] Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso.1’ 2% [==> ] 11,726,532 500KB/s eta 12m 31s
4. Do Not Download Multiple Versions
By default if you download the same file to the same directory for a second time, it will be given the name with a .1 appended to the end. The next instance will have .2, followed by .3 and so on.
This was seen above in our previous example, as we attempted to download the same .iso file again we can see that it was saved with the .1 extension.
Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso.1’
If this behaviour is undesirable we can instead us the -nc option for ‘no clobber’ which stops this, and prevents the downloading of newer copies of the file if they already exist.
[root@centos7 Downloads]# wget -nc https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso File ‘CentOS-7-x86_64-NetInstall-1511.iso’ already there; not retrieving.
5. Don’t Check SSL/TLS Certificate
By default when downloading over HTTPS wget will check the certificate against available CA’s, however if the certificate is bad or perhaps self signed the transfer will fail.
[root@centos7 Downloads]# wget https://localhost/file.txt --2016-08-31 11:44:14-- https://localhost/file.txt Resolving localhost (localhost)... ::1, 127.0.0.1 Connecting to localhost (localhost)|::1|:443... connected. ERROR: cannot verify localhost's certificate, issued by ‘/C=--/ST=SomeState/ L=SomeCity/O=SomeOrganization/OU=SomeOrganizationalUnit/ CN=centos7.example.com/[email protected]’: Unable to locally verify the issuer's authority. ERROR: certificate common name ‘centos7.example.com’ doesn't match requested host name ‘localhost’. To connect to localhost insecurely, use `--no-check-certificate'.
We may want to ignore this check and download the file anyway. This is done with the --no-check-certificate option, which forces an insecure mode of operation allowing you to proceed. Generally this is not a good choice when transferring confidential data, however it is quite useful in testing environments where self signed certificates are commonly used.
[root@centos7 Downloads]# wget https://localhost/file.txt --no-check-certificate --2016-08-31 11:44:32-- https://localhost/file.txt Resolving localhost (localhost)... ::1, 127.0.0.1 Connecting to localhost (localhost)|::1|:443... connected. WARNING: cannot verify localhost's certificate, issued by ‘/C=--/ST=SomeState/ L=SomeCity/O=SomeOrganization/OU=SomeOrganizationalUnit/ CN=centos7.example.com/[email protected]’: Unable to locally verify the issuer's authority. WARNING: certificate common name ‘centos7.example.com’ doesn't match requested host name ‘localhost’. HTTP request sent, awaiting response... 200 OK Length: 406 [text/plain] Saving to: ‘file.txt’ 100%[============================================================>] 406 --.-K/s in 0s 2016-08-31 11:44:32 (52.1 MB/s) - ‘file.txt’ saved [406/406]
6. Enable Timestamps
By default the timestamp of the locally downloaded file will be the same as the file on the remote server. We can instead use --no-use-server-timestamps which will not set the local files timestamp to the one provided by the file on the server, but instead be set to when the file was downloaded which may be more useful as it allows us to see when the file was downloaded.
As shown we have our original .iso file from the first example which is dated ‘Dec 10 2015′, however when we download our second copy with --no-use-server-timestamps specified once complete its timestamp is listed as today, ’31 Aug 2016’.
[root@centos7 Downloads]# ls -la total 477668 -rw-r--r--. 1 root root 394264576 Dec 10 2015 CentOS-7-x86_64-NetInstall-1511.iso [root@centos7 Downloads]# wget --no-use-server-timestamps https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso --2016-08-31 11:50:10-- https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 394264576 (376M) [application/octet-stream] Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso.1’ 100%[================================================================>] 394,264,576 1.21MB/s in 3m 44s 2016-08-31 11:53:55 (1.68 MB/s) - ‘CentOS-7-x86_64-NetInstall-1511.iso.1’ saved [394264576/394264576] [root@centos7 Downloads]#
ls -la total 770060 -rw-r--r--. 1 root root 394264576 Dec 10 2015 CentOS-7-x86_64-NetInstall-1511.iso -rw-r--r--. 1 root root 394264576 Aug 31 11:53 CentOS-7-x86_64-NetInstall-1511.iso.1 7. Change Progress Bar
By default wget will show a progress bar in the form of ‘=====>’ along with the percentage complete. We can change this with the --progress option to ‘dot’, as shown below.
[root@centos7 Downloads]# wget --progress=dot https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso --2016-08-31 11:47:07-- https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 394264576 (376M) [application/octet-stream] Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso.2’ 0K .......... .......... .......... .......... .......... 0% 374K 17m8s 50K .......... .......... .......... .......... .......... 0% 1.33M 10m55s 100K .......... .......... .......... .......... .......... 0% 744K 10m9s 150K .......... .......... .......... .......... .......... 0% 740K 9m47s 200K .......... .......... .......... .......... .......... 0% 1.37M 8m44s 250K .......... .......... .......... .......... .......... 0% 753K 8m42s 300K .......... .......... .......... .......... .......... 0% 1.39M 8m6s 350K .......... .......... .......... .......... .......... 0% 752K 8m9s 400K .......... .......... .......... .......... .......... 0% 1.38M 7m45s 450K .......... .......... .......... .......... .......... 0% 760K 7m49s 500K .......... .......... .......... .......... .......... 0% 735K 7m54s ...This is just a simple alternative we can use to display the output differently while downloading a file.
8. Continue Partially Downloaded File
If we have a partially downloaded file that did not fully complete, we can make use of the -c option to continue the download of a file from a previous instance of wget or some other program.
Essentially if we run wget with -c and we already have a file with the same name on our local system, wget will assume that the part of the file we already have is correct and use it as an offset when requesting the rest of the data. This way the server will only need to transfer the remaining part.
In the below example we started performing a normal wget for the .iso file, then we pressed ‘ctrl+c’ to cancel the transfer, leaving us with a partially complete file. Here we run wget again with -c to continue the transfer, we can see in the output that the part of the file we already have is denoted by the ‘+’ character.
[root@centos7 Downloads]# wget -c https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso --2016-08-31 11:33:26-- https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected. HTTP request sent, awaiting response... 206 Partial Content Length: 394264576 (376M), 74101684 (71M) remaining [application/octet-stream] Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso’ 94% [+++++++++++++++++++++++++++++++====> ] 372,277,944 2.12MB/s eta 8s
This should only be required to resume a download that was started in a previous instance of wget for local files that we still have partially completed. Without -c, the download would simply start again with a new .1 file, leaving the existing incomplete file as is.
9. Retry Failed Download
We can specify how many times a file transfer should retry with the -t option followed by the number of retries to attempt should there be a failure. The default is 20 times unless fatal errors like connection refused or 404 not found are encountered.
In the following example the network connection was intentionally disrupted, we can see that the retry takes place automatically and wget will keep trying.
[root@centos7 Downloads]# wget -t 10 https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso --2016-08-31 11:37:47-- https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 394264576 (376M) [application/octet-stream] Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso.1’ 22% [===============================> ] 86,849,288 504KB/s in 2m 50s 2016-08-31 11:40:37 (500 KB/s) - Connection closed at byte 86849288. Retrying. --2016-08-31 11:40:38-- (try: 2) https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 394264576 (376M) [application/octet-stream] Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso.1’ 0% [ ] 0 --.-K/s in 0.1s
Unlike continuing with -c, this only works for retrying within the current running instance of wget, rather than resuming a partial file from a previous instance.
10. Download From URL File List
We don’t have to specify the URL in each individual wget command, we can instead use -i followed by a file containing multiple URLs (one URL per line) and wget will go through and download them all.
[root@centos7 Downloads]# cat to-download.txt https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/0_README.txt https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt [root@centos7 Downloads]# wget -i to-download.txt --2016-08-31 12:03:15-- https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/0_README.txt Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 2498 (2.4K) [text/plain] Saving to: ‘0_README.txt’ 100%[======================================================>] 2,498 --.-K/s in 0.004s 2016-08-31 12:03:15 (611 KB/s) - ‘0_README.txt’ saved [2498/2498] --2016-08-31 12:03:15-- https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt Reusing existing connection to mirror.aarnet.edu.au:443. HTTP request sent, awaiting response... 200 OK Length: 406 [text/plain] Saving to: ‘md5sum.txt’ 100%[======================================================>] 406 --.-K/s in 0s 2016-08-31 12:03:16 (96.0 MB/s) - ‘md5sum.txt’ saved [406/406] FINISHED --2016-08-31 12:03:16-- Total wall clock time: 0.6s Downloaded: 2 files, 3.4K in 0.02s (207 KB/s)
11. Wait Time
By default the next retry will happen straight away, however we can delay it by specifying the -w wait option, which takes a number of seconds to wait before the next attempt. This can be useful if the remote server is slow or unresponsive, as we may not want to overload it with attempts.
[root@centos7 Downloads]# time wget -w 10 -i to-download.txt --2016-08-31 12:06:30-- https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/0_README.txt Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 2498 (2.4K) [text/plain] Saving to: ‘0_README.txt’ 100%[===============================================================>] 2,498 --.-K/s in 0s 2016-08-31 12:06:30 (675 MB/s) - ‘0_README.txt’ saved [2498/2498] --2016-08-31 12:06:40-- https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt Reusing existing connection to mirror.aarnet.edu.au:443. HTTP request sent, awaiting response... 200 OK Length: 406 [text/plain] Saving to: ‘md5sum.txt’ 100%[===============================================================>] 406 --.-K/s in 0s 2016-08-31 12:06:40 (143 MB/s) - ‘md5sum.txt’ saved [406/406] FINISHED --2016-08-31 12:06:40-- Total wall clock time: 10s Downloaded: 2 files, 2.8K in 0s (443 MB/s) real 0m10.337s user 0m0.020s sys 0m0.006s
In this example we download two files with a 10 second wait time between them. We can see that the total command execution time took just over 10 seconds as expected as we’re dealing with very small files here. The output also confirms that the first file was downloaded at 12:06:30, while the second file started downloading at 12:06:40 which is 10 seconds later as specified in our -w option.
Rather than having wait times on each file download, we can instead specify to only wait between retries of failed downloads. This is done with --waitretry followed by the number of seconds to wait. This has a default of 10 seconds and will start with waiting 1 second after the first failure, followed by 2 seconds, increasing continually up to the specified number.
We can also specify a random wait time with --random-wait followed by a number of seconds. This can help prevent a web server detecting obvious constant file retrieval as the accesses will now be randomized.
12. Non-Interactive Download
Up until now we have been downloading with wget in the foreground. Wget is a non-interactive downloader, which allows us to run it in the background without a user session. This is done with the -b option for background transfers, and creates a wget-log file with updates as they happen.
[root@centos7 Downloads]# wget -b https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso Continuing in background, pid 2789. Output will be written to ‘wget-log’. [root@centos7 Downloads]# tailf wget-log 14950K .......... .......... .......... .......... .......... 3% 1.52M 2m12s 15000K .......... .......... .......... .......... .......... 3% 12.5M 2m12s 15050K .......... .......... .......... .......... .......... 3% 1.47M 2m12s 15100K .......... .......... .......... .......... .......... 3% 1.54M 2m12s 15150K .......... .......... .......... .......... .......... 3% 11.5M 2m12s
As shown our wget transfer is sent to the background straight away, giving us our bash shell back. We can then view the wget-log file which logs information on the status of the transfer.
13. Send Messages To Log File
Rather than having messages output to the screen via stdout/stderr, we can use -o to log all messages to the file specified.
[root@centos7 Downloads]# wget -o /root/wget-log.txt https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
[root@centos7 Downloads]# cat /root/wget-log.txt
--2016-08-31 12:18:26-- https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef
Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 406 [text/plain]
Saving to: ‘md5sum.txt.1’
0K 100% 4.60M=0s
2016-08-31 12:18:27 (4.60 MB/s) - ‘md5sum.txt.1’ saved [406/406]
It’s important to note that -o will overwrite an existing file, we can instead use -a which works in the same way, but will append to the end of a file if it already exists.
14. Hide Output
Instead if we want no output we can use the -q option for quiet mode which will hide all output.
[root@centos7 Downloads]# wget -q https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt [root@centos7 Downloads]#
15. Debug Information
More information can be obtained with the debug option, -d which may be useful when troubleshooting any problems.
[root@centos7 Downloads]# wget -d https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt DEBUG output created by Wget 1.14 on linux-gnu. URI encoding = ‘UTF-8’ --2016-08-31 13:31:24-- https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef Caching mirror.aarnet.edu.au => 202.158.214.106 2001:388:30bc:cafe::beef Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected. Created socket 3. Releasing 0x0000000002465f50 (new refcount 1). Initiating SSL handshake. Handshake successful; connected socket 3 to SSL handle 0x000000000247adb0 certificate: subject: /1.3.6.1.4.1.311.60.2.1.3=AU/businessCategory=Non-Commercial Entity/ serialNumber=54 084 540 518/C=AU/ST=New South Wales/L=North Ryde/O=AARNET Pty Ltd/OU=Operations/CN=mirror.aarnet.edu.au issuer: /C=BM/O=QuoVadis Limited/CN=QuoVadis EV SSL ICA G1 X509 certificate successfully verified and matches host mirror.aarnet.edu.au ---request begin--- GET /pub/centos/7/isos/x86_64/md5sum.txt HTTP/1.1 User-Agent: Wget/1.14 (linux-gnu) Accept: */* Host: mirror.aarnet.edu.au Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Wed, 31 Aug 2016 03:30:16 GMT Server: ATS/5.2.0 Last-Modified: Thu, 10 Dec 2015 15:37:35 GMT Accept-Ranges: bytes Content-Length: 406 Content-Type: text/plain; charset=UTF-8 Age: 96 ---response end--- 200 OK Registered socket 3 for persistent reuse. URI content encoding = ‘UTF-8’ Length: 406 [text/plain] Saving to: ‘md5sum.txt.4’ 100%[=====================================================>] 406 --.-K/s in 0s 2016-08-31 13:31:24 (126 MB/s) - ‘md5sum.txt.4’ saved [406/406]
Here we can see all sorts of information about the certificate and HTTP request.
By default all wget requests are run with the -v option for verbose output, however we can run -nv for no verbose to cut down on the output and only receive the bare essentials.
[root@centos7 Downloads]# wget -nv https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt 2016-08-31 13:33:43 URL:https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt [406/406] -> "md5sum.txt.7" [1]
16. View Server Response
Similarly we can also print just the HTTP headers sent by the server or responses sent by FTP servers with the -S option for server responses, which may be useful when troubleshooting as it allows us to see what the remote server is saying in response to our requests. This information is also present in the much more verbose debug logging shown above.
[root@centos7 Downloads]# wget -S https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt --2016-08-31 13:37:00-- https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK Date: Wed, 31 Aug 2016 03:37:29 GMT Server: ATS/5.2.0 Last-Modified: Thu, 10 Dec 2015 15:37:35 GMT Accept-Ranges: bytes Content-Length: 406 Content-Type: text/plain; charset=UTF-8 Age: 0 Length: 406 [text/plain] Saving to: ‘md5sum.txt.8’ 100%[===================================================================================================================================================>] 406 --.-K/s in 0s 2016-08-31 13:37:00 (83.8 MB/s) - ‘md5sum.txt.8’ saved [406/406]
17. Timeout
If a download takes too long, we can abort and cancel it automatically after a total time period specified by the -T option followed by the number of seconds we want to timeout after.
[root@centos7 Downloads]# wget -T 30 https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
The default timeout of 900 seconds of idle where no data is received during a download is usually fine, however it can be adjusted as per your requirements.
18. Use Credentials
If a HTTP or FTP connection requires user authentication we can specify a username with the --user option and the password with the --password option. Generally speaking it’s not a good idea in terms of security to use the --password option, as our specified password will be stored in the bash history. We can instead use --ask-password which will prompt for the password, keeping it out of our history log.
[root@centos7 Downloads]# wget --user=username --ask-password http://localhost/file.txt Password for user ‘username’:
19. Proxy Authentication
While the above example is used for credentials at the remote host, we can also specify a proxy server and optionally proxy credentials if we have a proxy server in our network that we need to authenticate against before being allowed to proxy through it.
This is done by first setting http_proxy, https_proxy, or ftp_proxy environment variables so that wget knows where the proxy server can be found.
[root@centos7 Downloads]# http_proxy=http://proxy.example.com:3128 [root@centos7 Downloads]# https_proxy=http://proxy.example.com:3128 [root@centos7 Downloads]# echo $http_proxy http://proxy.example.com:3128 [root@centos7 Downloads]# echo $https_proxy http://proxy.example.com:3128
Now wget will be aware of the proxy specified.
If the proxy requires user authentication we can then specify --proxy-user and --proxy-password options to authenticate and proceed.
20. Download Non Cached File
In a HTTP request we can specify --no-cache which will send the ‘Pragma: no-cache’ and ‘Cache-Control: no-cache, must-revalidate’ options to the web server, requesting a non cached version which may be useful if you want to download the most up to date version of a file and skip the cache. By default accepting cached files is allowed.
With the use of -d, we can see our HTTP request contains ‘Pragma: no-cache’ and ‘Cache-Control: no-cache, must-revalidate’ only after adding --no-cache.
[root@centos7 Downloads]# wget -d https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt ... ---request begin--- GET /pub/centos/7/isos/x86_64/md5sum.txt HTTP/1.1 User-Agent: Wget/1.14 (linux-gnu) Accept: */* Host: mirror.aarnet.edu.au Connection: Keep-Alive ---request end--- [root@centos7 Downloads]# wget -d --no-cache https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt ... ---request begin--- GET /pub/centos/7/isos/x86_64/md5sum.txt HTTP/1.1 Cache-Control: no-cache, must-revalidate Pragma: no-cache User-Agent: Wget/1.14 (linux-gnu) Accept: */* Host: mirror.aarnet.edu.au Connection: Keep-Alive ---request end---
Summary
We have seen that while the wget command can be used to simply download a remote file to the local Linux system, it also has plenty of other advanced features available which make it a very powerful command worth learning.
0 Comments.