It would seem that in a world where half of the world's population is actively using computers and is used to stylish graphical interfaces and its rich features, there is no longer room for text applications. Some 20 years ago, the computer interface was a terrible black panel that did not display anything but a set of white characters, and all control was carried out through text commands. Surprisingly, just as ugly these programs are, they are just as functional and reliable.
What is Wget?
Wget (also called GNU Wget Linux) is a console, non-interactive file and web page application. In fact, this is a program for downloading files from the Internet, characterized by the lack of a graphical interface and a number of management features that are not typical for more modern solutions.
The program was developed by an engineer from Croatia, Hrvoji Niksic, and now the development is supported by Giuseppe Scrivano and the Linux community. The first build of the utility (then called GetURL) was introduced in January 1996. The program was written in the C programming language and published under the GNU license. The tool was originally released on the Linux platform. Over time, the utility was translated into many languages and ported to all popular platforms, including Wget for Windows 7 (and newer), Wget for MacOS, and so on.
This program is not interactive, which means that any process launched within its capabilities cannot be taken under control, it can be controlled by contacting the controls of the operating system itself.
Despite the fact that modern browsers usually have a built-in download manager, they are nevertheless designed for interactive interaction, so working in a manual mode becomes difficult. Wget, by contrast, allows a number of processes to be automated. For example, it is possible to pre-prepare a text document with a list of web pages, using which you can download several files and pages at once.
The main functions of the program
Nevertheless, first of all, this is a program for downloading files, from which the obviousness of its main function follows.
Wget can download any materials from the Internet, whether it be documents, executive files or entire web pages in HTML format (using various protocols). Files can be downloaded from directories via FTP.
The client can perform recursive loading, automatically following the links within a specific website (or several websites), while restoring the original structure of the web page. It is possible to view downloaded websites without access to the network (by default it is loaded in the form of addresses for online viewing).
Initially, Wget adapted to slow and unstable connection types, so its important feature is the ability to restore interrupted file downloads (this is possible if the server supports RANGE headers). Wget can also monitor the status of servers (their availability) for reloading and updating (Wget checks file headers and supports the latest version) of files and web pages.
The program also has a number of features typical of the so-called spider utilities, which examine various websites and servers for the presence of certain files on them.
Supported Platforms
Wget Linux has become the base for this product. As mentioned above, initially the utility worked only on Linux, but quickly gained popularity and was adapted for other systems. For example, you can find the version of Wget for Windows 7 and more modern operating systems, and there are assemblies with a graphical interface. As for Linux, there the program is often sewn into the distribution. Terminals in some Linux builds natively support Wget commands. Ubuntu, Fedora, openSUSE and other consumer distributions work with the program, as they say, out of the box. Next, we consider the program in more detail.
Wget Commands: List of Key Parameters
The basic list of commands includes:
- -V - allows you to find out which version of the program is currently being used;
- -h - shows the most complete list of commands for working with the application;
- -b - allows you to put the program in the background;
Message Options List
Wget-commands allow you to fully control the displayed information. Here is a list of them:
- -o logfile - creates a log file in which the entire process of the application is recorded;
- -a logfile - allows to supplement the existing log file instead of creating a new one;
- -d - displays the information necessary for developers and the debugger (helps to solve problems and fix errors in the client);
- -q - disables the display of messages;
- -v - this option is enabled by default, provides a detailed report on all processes occurring in the program (in real time);
- -nv - significantly reduces the information received by the user in messages. Only the most necessary data and error alerts remain;
- -i file (with the path to the file) - the command allows you to scan the file, extracting the link from it and uploading; you can use files of various formats, including HTML, but if you use this command, you should not specify the path to the file on the command line;
- -f - provokes a program to read a file as HTML; when using this command to work with a local HTML document, an error may occur to avoid it, just specify in the document <base href = ”url”>.
Download Options List
Basic commands for working with file downloads using Wget. Here is their basic set:
- --bind-address = ADDRESS - this command allows you to transfer bind data to ADDRESS on the local machine. When using a TCP / IP connection, the command is necessary to specify the IP address (if there are several).
- -t number (you need to specify a number) - allows the user to choose the number of retries To avoid repetitions, you should specify the value number equal to zero.
- -nc - this command allows you to return to the previous state of the downloaded file if there was a network failure, and the download was interrupted. If you do not use one of the commands to reload (-N, -nc, -r), then Wget will create a copy of it and start downloading again. The -r option replaces existing files.
- -c - also helps to reload the file in the event of a loss of connection (and when Wget stopped making independent attempts to restore the connection). It is enough to indicate this command before the link to the file. If there is a similar file in the directory used, then Wget will identify it and automatically decide on how to reload it. The command can only be used on servers that have RANGE header support.
- --progress - allows you to select the type of display of the progress of the download. You can specify –progress = bar or progress = dot.
- --spider - replaces the file upload command with a check for its presence on the server;
- -w (in seconds) - indicates the time interval between downloads.
- -q (indicated in the number of kilobytes or megabytes) - allows you to clearly adjust the size of the downloaded data and limit excessive volumes. Nevertheless, it is worth noting that one file will be downloaded regardless of the quota set (that is, this option will not work with one file, no matter how heavy it is).
HTTP Settings List
- --http-user = and --http-passwd = - allow you to specify a login and password for authorization on the web resource from which the files will be downloaded.
- -C = on / off - allows you to enable or disable data caching on the server side.
- --cookies = on / off - the same as for cache, only for cookies. Allows you to either stop or resume the exchange of cookie data necessary for the server to maintain statistics on visits. By default, this option is enabled, but cookies are not saved to the hard drive.
- --save-cookies - allows you to save cookies (only recent data).
- --proxy-user and –proxy-passwd - allows you to specify the login and password for authorization on the proxy server (if one is involved).
FTP Settings List
Advanced options for working with Wget. FTP Commands:
- -nr - prohibits the removal of temporary files from the .listing directory, which are generated spontaneously when working with FTP;
- -g on / off - allows the utility to use special characters in FTP directories;
- --passive-ftp - this command can be used if there is a Firewall to activate passive FTP mode (when the user is responsible for establishing a connection to the server).
Wget Commands: Case Studies
The simplest and most common example for working with the program is, of course, downloading files. To test Wget commands, you can start with a simple one, find a link to some file on the network and enter the / terminal: Wget * link in the console indicating the exact and full path to the file *.
To continue downloading the file, use: Wget -c * link indicating the exact and full path to the file *.
If there is a planned list with links to files and documents, then you can use the Wget -i filelist.txt command.
As for the loading of web pages, everything is somewhat more complicated, but at the same time the number of possibilities increases significantly. In order not to re-enter the parameters every time when it becomes necessary to download the website, you can create scripts or write all the settings to the .wgetrc file.
You can save the web page as links (it will be difficult to open them on the local machine) using the Wget -m * link command with the exact and full path to the file *.
To make the same page available for viewing without connecting to the Internet, you can use the Wget -r -10 -k * link command with the exact and full path to the file *.
Wget format error command not found can mean any typo and incorrectly specified parameter.