20 wget Command Examples For Linux

Wget is a non-interactive network downloader which can be used for downloading files in Unix/Linux. It supports the HTTP, HTTPS and FTP protocols and also has proxy support.

The wget command is quite powerful and has a lot of options available, in this guide we’ll be covering 20 of the most important wget examples that will help you learn how to best use it.

Install Wget

It is not uncommon for wget to not be installed by default, if the command is not available for you then you must first install it as shown below.

CentOS/RHEL/Fedora

[[email protected] ~]# yum install wget -y

Debian/Ubuntu

[email protected]:~# apt-get install wget -y

How To use wget – Command Examples

Now that we have wget installed and ready to use, here are our top wget command examples.

  • 1. Download A File

    Lets begin with a basic example of downloading a simple file from the Internet. We simply specify the file that we want to download after the wget command, as shown below.

    [[email protected] Downloads]# wget https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    --2016-08-31 11:30:54--  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef
    Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 394264576 (376M) [application/octet-stream]
    Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso’
    
    68% [=========================>             ] 271,936,588 2.32MB/s  eta 54s
    

    The output provides us with percentage complete, download speed, and time remaining estimate.

    This has downloaded a copy of the file to the current working directory with the same file name as was in the URL.

    [[email protected] Downloads]# ls
    CentOS-7-x86_64-NetInstall-1511.iso
    
  • 2. Download To Specific Directory With New Name

    We can use the -O option to specify the output file, allowing us to tell wget where to download the file to, and what to name the file. This saves us having to download it to the current working directory, move it, and then potentially rename it as may be required if following the first example.

    [[email protected] Downloads]# wget -O /root/file.txt https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    --2016-08-31 11:36:25--  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef
    Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 406 [text/plain]
    Saving to: ‘/root/file.txt’
    
    100%[==============================================================================>] 406         --.-K/s   in 0s
    
    2016-08-31 11:36:25 (139 MB/s) - ‘/root/file.txt’ saved [406/406]
    

    Here we have downloaded the remote file md5sum.txt and saved it to /root/file.txt.

  • 3. Limit Download Speed

    Rather than letting wget download at maximum speed which is the default, we can optionally cap it at a specific speed with the --limit-rate option. This may be useful if you don’t want wget to use all available network bandwidth.

    The speed limit can be defined in a number of bytes, or kilobytes with the k suffix, or megabytes with the m suffix.

    In the below example, we start downloading the .ISO file with a 500KB/s limit, and this is confirmed in the wget output.

    [[email protected] Downloads]# wget --limit-rate=500k https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    --2016-08-31 11:37:47--  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef
    Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 394264576 (376M) [application/octet-stream]
    Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso.1’
    
     2% [==>                                              ] 11,726,532   500KB/s  eta 12m 31s
    
  • 4. Do Not Download Multiple Versions

    By default if you download the same file to the same directory for a second time, it will be given the name with a .1 appended to the end. The next instance will have .2, followed by .3 and so on.

    This was seen above in our previous example, as we attempted to download the same .iso file again we can see that it was saved with the .1 extension.

    Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso.1

    If this behaviour is undesirable we can instead us the -nc option for ‘no clobber’ which stops this, and prevents the downloading of newer copies of the file if they already exist.

    [[email protected] Downloads]# wget -nc https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    File ‘CentOS-7-x86_64-NetInstall-1511.iso’ already there; not retrieving.
    
  • 5. Don’t Check SSL/TLS Certificate

    By default when downloading over HTTPS wget will check the certificate against available CA’s, however if the certificate is bad or perhaps self signed the transfer will fail.

    [[email protected] Downloads]# wget https://localhost/file.txt
    --2016-08-31 11:44:14--  https://localhost/file.txt
    Resolving localhost (localhost)... ::1, 127.0.0.1
    Connecting to localhost (localhost)|::1|:443... connected.
    ERROR: cannot verify localhost's certificate, issued by ‘/C=--/ST=SomeState/
    L=SomeCity/O=SomeOrganization/OU=SomeOrganizationalUnit/
    [email protected]:
      Unable to locally verify the issuer's authority.
        ERROR: certificate common name ‘centos7.example.com’ doesn't match requested host name ‘localhost’.
    To connect to localhost insecurely, use `--no-check-certificate'.
    

    We may want to ignore this check and download the file anyway. This is done with the --no-check-certificate option, which forces an insecure mode of operation allowing you to proceed. Generally this is not a good choice when transferring confidential data, however it is quite useful in testing environments where self signed certificates are commonly used.

    [[email protected] Downloads]# wget https://localhost/file.txt --no-check-certificate
    --2016-08-31 11:44:32--  https://localhost/file.txt
    Resolving localhost (localhost)... ::1, 127.0.0.1
    Connecting to localhost (localhost)|::1|:443... connected.
    WARNING: cannot verify localhost's certificate, issued by ‘/C=--/ST=SomeState/
    L=SomeCity/O=SomeOrganization/OU=SomeOrganizationalUnit/
    [email protected]:
      Unable to locally verify the issuer's authority.
        WARNING: certificate common name ‘centos7.example.com’ doesn't match requested host name ‘localhost’.
    HTTP request sent, awaiting response... 200 OK
    Length: 406 [text/plain]
    Saving to: ‘file.txt’
    
    100%[============================================================>] 406         --.-K/s   in 0s
    
    2016-08-31 11:44:32 (52.1 MB/s) - ‘file.txt’ saved [406/406]
    
  • 6. Enable Timestamps

    By default the timestamp of the locally downloaded file will be the same as the file on the remote server. We can instead use --no-use-server-timestamps which will not set the local files timestamp to the one provided by the file on the server, but instead be set to when the file was downloaded which may be more useful as it allows us to see when the file was downloaded.

    As shown we have our original .iso file from the first example which is dated ‘Dec 10 2015′, however when we download our second copy with --no-use-server-timestamps specified once complete its timestamp is listed as today, ’31 Aug 2016’.

    [[email protected] Downloads]# ls -la
    total 477668
    -rw-r--r--. 1 root root 394264576 Dec 10  2015 CentOS-7-x86_64-NetInstall-1511.iso
    
    [[email protected] Downloads]# wget --no-use-server-timestamps https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    --2016-08-31 11:50:10--  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef
    Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 394264576 (376M) [application/octet-stream]
    Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso.1’
    
    100%[================================================================>] 394,264,576 1.21MB/s   in 3m 44s
    
    2016-08-31 11:53:55 (1.68 MB/s) - ‘CentOS-7-x86_64-NetInstall-1511.iso.1’ saved [394264576/394264576]
    
    [[email protected] Downloads]# ls -la
    total 770060
    -rw-r--r--. 1 root root 394264576 Dec 10  2015 CentOS-7-x86_64-NetInstall-1511.iso
    -rw-r--r--. 1 root root 394264576 Aug 31 11:53 CentOS-7-x86_64-NetInstall-1511.iso.1
    
  • 7. Change Progress Bar

    By default wget will show a progress bar in the form of ‘=====>’ along with the percentage complete. We can change this with the --progress option to ‘dot’, as shown below.

    [[email protected] Downloads]# wget --progress=dot https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    --2016-08-31 11:47:07--  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef
    Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 394264576 (376M) [application/octet-stream]
    Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso.2’
    
         0K .......... .......... .......... .......... ..........  0%  374K 17m8s
        50K .......... .......... .......... .......... ..........  0% 1.33M 10m55s
       100K .......... .......... .......... .......... ..........  0%  744K 10m9s
       150K .......... .......... .......... .......... ..........  0%  740K 9m47s
       200K .......... .......... .......... .......... ..........  0% 1.37M 8m44s
       250K .......... .......... .......... .......... ..........  0%  753K 8m42s
       300K .......... .......... .......... .......... ..........  0% 1.39M 8m6s
       350K .......... .......... .......... .......... ..........  0%  752K 8m9s
       400K .......... .......... .......... .......... ..........  0% 1.38M 7m45s
       450K .......... .......... .......... .......... ..........  0%  760K 7m49s
       500K .......... .......... .......... .......... ..........  0%  735K 7m54s
       ...
    

    This is just a simple alternative we can use to display the output differently while downloading a file.

  • 8. Continue Partially Downloaded File

    If we have a partially downloaded file that did not fully complete, we can make use of the -c option to continue the download of a file from a previous instance of wget or some other program.

    Essentially if we run wget with -c and we already have a file with the same name on our local system, wget will assume that the part of the file we already have is correct and use it as an offset when requesting the rest of the data. This way the server will only need to transfer the remaining part.

    In the below example we started performing a normal wget for the .iso file, then we pressed ‘ctrl+c’ to cancel the transfer, leaving us with a partially complete file. Here we run wget again with -c to continue the transfer, we can see in the output that the part of the file we already have is denoted by the ‘+’ character.

    [[email protected] Downloads]# wget -c https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    --2016-08-31 11:33:26--  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef
    Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 394264576 (376M), 74101684 (71M) remaining [application/octet-stream]
    Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso’
    
    94% [+++++++++++++++++++++++++++++++====>   ] 372,277,944 2.12MB/s  eta 8s
    

    This should only be required to resume a download that was started in a previous instance of wget for local files that we still have partially completed. Without -c, the download would simply start again with a new .1 file, leaving the existing incomplete file as is.

  • 9. Retry Failed Download

    We can specify how many times a file transfer should retry with the -t option followed by the number of retries to attempt should there be a failure. The default is 20 times unless fatal errors like connection refused or 404 not found are encountered.

    In the following example the network connection was intentionally disrupted, we can see that the retry takes place automatically and wget will keep trying.

    [[email protected] Downloads]# wget -t 10 https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    --2016-08-31 11:37:47--  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef
    Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 394264576 (376M) [application/octet-stream]
    Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso.1’
    
    22% [===============================>                                          ] 86,849,288   504KB/s   in 2m 50s
    
    2016-08-31 11:40:37 (500 KB/s) - Connection closed at byte 86849288. Retrying.
    
    --2016-08-31 11:40:38--  (try: 2)  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 394264576 (376M) [application/octet-stream]
    Saving to: ‘CentOS-7-x86_64-NetInstall-1511.iso.1’
    
     0% [                                                                          ] 0           --.-K/s   in 0.1s
    

    Unlike continuing with -c, this only works for retrying within the current running instance of wget, rather than resuming a partial file from a previous instance.

  • 10. Download From URL File List

    We don’t have to specify the URL in each individual wget command, we can instead use -i followed by a file containing multiple URLs (one URL per line) and wget will go through and download them all.

    [[email protected] Downloads]# cat to-download.txt
    https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/0_README.txt
    https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    
    [[email protected] Downloads]# wget -i to-download.txt
    --2016-08-31 12:03:15--  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/0_README.txt
    Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef
    Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 2498 (2.4K) [text/plain]
    Saving to: ‘0_README.txt’
    
    100%[======================================================>] 2,498       --.-K/s   in 0.004s
    
    2016-08-31 12:03:15 (611 KB/s) - ‘0_README.txt’ saved [2498/2498]
    
    --2016-08-31 12:03:15--  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    Reusing existing connection to mirror.aarnet.edu.au:443.
    HTTP request sent, awaiting response... 200 OK
    Length: 406 [text/plain]
    Saving to: ‘md5sum.txt’
    
    100%[======================================================>] 406         --.-K/s   in 0s
    
    2016-08-31 12:03:16 (96.0 MB/s) - ‘md5sum.txt’ saved [406/406]
    
    FINISHED --2016-08-31 12:03:16--
    Total wall clock time: 0.6s
    Downloaded: 2 files, 3.4K in 0.02s (207 KB/s)
    
  • 11. Wait Time

  • By default the next retry will happen straight away, however we can delay it by specifying the -w wait option, which takes a number of seconds to wait before the next attempt. This can be useful if the remote server is slow or unresponsive, as we may not want to overload it with attempts.

    [[email protected] Downloads]# time wget -w 10 -i to-download.txt
    --2016-08-31 12:06:30--  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/0_README.txt
    Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef
    Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 2498 (2.4K) [text/plain]
    Saving to: ‘0_README.txt’
    
    100%[===============================================================>] 2,498       --.-K/s   in 0s
    
    2016-08-31 12:06:30 (675 MB/s) - ‘0_README.txt’ saved [2498/2498]
    
    --2016-08-31 12:06:40--  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    Reusing existing connection to mirror.aarnet.edu.au:443.
    HTTP request sent, awaiting response... 200 OK
    Length: 406 [text/plain]
    Saving to: ‘md5sum.txt’
    
    100%[===============================================================>] 406         --.-K/s   in 0s
    
    2016-08-31 12:06:40 (143 MB/s) - ‘md5sum.txt’ saved [406/406]
    
    FINISHED --2016-08-31 12:06:40--
    Total wall clock time: 10s
    Downloaded: 2 files, 2.8K in 0s (443 MB/s)
    
    real    0m10.337s
    user    0m0.020s
    sys     0m0.006s
    

    In this example we download two files with a 10 second wait time between them. We can see that the total command execution time took just over 10 seconds as expected as we’re dealing with very small files here. The output also confirms that the first file was downloaded at 12:06:30, while the second file started downloading at 12:06:40 which is 10 seconds later as specified in our -w option.

    Rather than having wait times on each file download, we can instead specify to only wait between retries of failed downloads. This is done with --waitretry followed by the number of seconds to wait. This has a default of 10 seconds and will start with waiting 1 second after the first failure, followed by 2 seconds, increasing continually up to the specified number.

    We can also specify a random wait time with --random-wait followed by a number of seconds. This can help prevent a web server detecting obvious constant file retrieval as the accesses will now be randomized.

  • 12. Non-Interactive Download

    Up until now we have been downloading with wget in the foreground. Wget is a non-interactive downloader, which allows us to run it in the background without a user session. This is done with the -b option for background transfers, and creates a wget-log file with updates as they happen.

    [[email protected] Downloads]# wget -b https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/CentOS-7-x86_64-NetInstall-1511.iso
    Continuing in background, pid 2789.
    Output will be written to ‘wget-log’.
    
    [[email protected] Downloads]# tailf wget-log
     14950K .......... .......... .......... .......... ..........  3% 1.52M 2m12s
     15000K .......... .......... .......... .......... ..........  3% 12.5M 2m12s
     15050K .......... .......... .......... .......... ..........  3% 1.47M 2m12s
     15100K .......... .......... .......... .......... ..........  3% 1.54M 2m12s
     15150K .......... .......... .......... .......... ..........  3% 11.5M 2m12s
    

    As shown our wget transfer is sent to the background straight away, giving us our bash shell back. We can then view the wget-log file which logs information on the status of the transfer.

  • 13. Send Messages To Log File

    Rather than having messages output to the screen via stdout/stderr, we can use -o to log all messages to the file specified.

    [[email protected] Downloads]# wget -o /root/wget-log.txt https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    [[email protected] Downloads]# cat /root/wget-log.txt
    --2016-08-31 12:18:26--  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef
    Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 406 [text/plain]
    Saving to: ‘md5sum.txt.1’
    
         0K                                                       100% 4.60M=0s
    
    2016-08-31 12:18:27 (4.60 MB/s) - ‘md5sum.txt.1’ saved [406/406]
    

    It’s important to note that -o will overwrite an existing file, we can instead use -a which works in the same way, but will append to the end of a file if it already exists.

  • 14. Hide Output

    Instead if we want no output we can use the -q option for quiet mode which will hide all output.

    [[email protected] Downloads]# wget -q https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    [[email protected] Downloads]#
    
  • 15. Debug Information

    More information can be obtained with the debug option, -d which may be useful when troubleshooting any problems.

    [[email protected] Downloads]# wget -d https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    DEBUG output created by Wget 1.14 on linux-gnu.
    
    URI encoding = ‘UTF-8’
    --2016-08-31 13:31:24--  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef
    Caching mirror.aarnet.edu.au => 202.158.214.106 2001:388:30bc:cafe::beef
    Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected.
    Created socket 3.
    Releasing 0x0000000002465f50 (new refcount 1).
    Initiating SSL handshake.
    Handshake successful; connected socket 3 to SSL handle 0x000000000247adb0
    certificate:
      subject: /1.3.6.1.4.1.311.60.2.1.3=AU/businessCategory=Non-Commercial Entity/
      serialNumber=54 084 540 518/C=AU/ST=New South Wales/L=North Ryde/O=AARNET 
      Pty Ltd/OU=Operations/CN=mirror.aarnet.edu.au
      issuer:  /C=BM/O=QuoVadis Limited/CN=QuoVadis EV SSL ICA G1
    X509 certificate successfully verified and matches host mirror.aarnet.edu.au
    
    ---request begin---
    GET /pub/centos/7/isos/x86_64/md5sum.txt HTTP/1.1
    User-Agent: Wget/1.14 (linux-gnu)
    Accept: */*
    Host: mirror.aarnet.edu.au
    Connection: Keep-Alive
    
    ---request end---
    HTTP request sent, awaiting response...
    ---response begin---
    HTTP/1.1 200 OK
    Date: Wed, 31 Aug 2016 03:30:16 GMT
    Server: ATS/5.2.0
    Last-Modified: Thu, 10 Dec 2015 15:37:35 GMT
    Accept-Ranges: bytes
    Content-Length: 406
    Content-Type: text/plain; charset=UTF-8
    Age: 96
    
    ---response end---
    200 OK
    Registered socket 3 for persistent reuse.
    URI content encoding = ‘UTF-8’
    Length: 406 [text/plain]
    Saving to: ‘md5sum.txt.4’
    
    100%[=====================================================>] 406         --.-K/s   in 0s
    
    2016-08-31 13:31:24 (126 MB/s) - ‘md5sum.txt.4’ saved [406/406]
    

    Here we can see all sorts of information about the certificate and HTTP request.

    By default all wget requests are run with the -v option for verbose output, however we can run -nv for no verbose to cut down on the output and only receive the bare essentials.

    [[email protected] Downloads]# wget -nv https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    2016-08-31 13:33:43 URL:https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt [406/406] -> "md5sum.txt.7" [1]
    
  • 16. View Server Response

    Similarly we can also print just the HTTP headers sent by the server or responses sent by FTP servers with the -S option for server responses, which may be useful when troubleshooting as it allows us to see what the remote server is saying in response to our requests. This information is also present in the much more verbose debug logging shown above.

    [[email protected] Downloads]# wget -S https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    --2016-08-31 13:37:00--  https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    Resolving mirror.aarnet.edu.au (mirror.aarnet.edu.au)... 202.158.214.106, 2001:388:30bc:cafe::beef
    Connecting to mirror.aarnet.edu.au (mirror.aarnet.edu.au)|202.158.214.106|:443... connected.
    HTTP request sent, awaiting response...
      HTTP/1.1 200 OK
      Date: Wed, 31 Aug 2016 03:37:29 GMT
      Server: ATS/5.2.0
      Last-Modified: Thu, 10 Dec 2015 15:37:35 GMT
      Accept-Ranges: bytes
      Content-Length: 406
      Content-Type: text/plain; charset=UTF-8
      Age: 0
    Length: 406 [text/plain]
    Saving to: ‘md5sum.txt.8’
    
    100%[===================================================================================================================================================>] 406         --.-K/s   in 0s
    
    2016-08-31 13:37:00 (83.8 MB/s) - ‘md5sum.txt.8’ saved [406/406]
    
  • 17. Timeout

    If a download takes too long, we can abort and cancel it automatically after a total time period specified by the -T option followed by the number of seconds we want to timeout after.

    [[email protected] Downloads]# wget -T 30 https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    

    The default timeout of 900 seconds of idle where no data is received during a download is usually fine, however it can be adjusted as per your requirements.

  • 18. Use Credentials

    If a HTTP or FTP connection requires user authentication we can specify a username with the --user option and the password with the --password option. Generally speaking it’s not a good idea in terms of security to use the --password option, as our specified password will be stored in the bash history. We can instead use --ask-password which will prompt for the password, keeping it out of our history log.

    [[email protected] Downloads]# wget --user=username --ask-password http://localhost/file.txt
    Password for user ‘username’:
    
  • 19. Proxy Authentication

    While the above example is used for credentials at the remote host, we can also specify a proxy server and optionally proxy credentials if we have a proxy server in our network that we need to authenticate against before being allowed to proxy through it.

    This is done by first setting http_proxy, https_proxy, or ftp_proxy environment variables so that wget knows where the proxy server can be found.

    [[email protected] Downloads]# http_proxy=http://proxy.example.com:3128
    [[email protected] Downloads]# https_proxy=http://proxy.example.com:3128
    [[email protected] Downloads]# echo $http_proxy
    http://proxy.example.com:3128
    [[email protected] Downloads]# echo $https_proxy
    http://proxy.example.com:3128
    

    Now wget will be aware of the proxy specified.

    If the proxy requires user authentication we can then specify --proxy-user and --proxy-password options to authenticate and proceed.

  • 20. Download Non Cached File

    In a HTTP request we can specify --no-cache which will send the ‘Pragma: no-cache’ and ‘Cache-Control: no-cache, must-revalidate’ options to the web server, requesting a non cached version which may be useful if you want to download the most up to date version of a file and skip the cache. By default accepting cached files is allowed.

    With the use of -d, we can see our HTTP request contains ‘Pragma: no-cache’ and ‘Cache-Control: no-cache, must-revalidate’ only after adding --no-cache.

    [[email protected] Downloads]# wget -d https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    ...
    ---request begin---
    GET /pub/centos/7/isos/x86_64/md5sum.txt HTTP/1.1
    User-Agent: Wget/1.14 (linux-gnu)
    Accept: */*
    Host: mirror.aarnet.edu.au
    Connection: Keep-Alive
    ---request end---
    
    [[email protected] Downloads]# wget -d --no-cache https://mirror.aarnet.edu.au/pub/centos/7/isos/x86_64/md5sum.txt
    ...
    ---request begin---
    GET /pub/centos/7/isos/x86_64/md5sum.txt HTTP/1.1
    Cache-Control: no-cache, must-revalidate
    Pragma: no-cache
    User-Agent: Wget/1.14 (linux-gnu)
    Accept: */*
    Host: mirror.aarnet.edu.au
    Connection: Keep-Alive
    ---request end---
    

Summary

We have seen that while the wget command can be used to simply download a remote file to the local Linux system, it also has plenty of other advanced features available which make it a very powerful command worth learning.

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>