Curl downloading multiple files

The most basic operation that Wget offers to users is downloading files by simply using its URL. This can be done by inputting the following command into the terminal:. Let us show an example to further clarify this. We will be downloading a simple image in the png format from the internet.

See the image below for better understanding:. Wget also allows users to download multiple files from different URLs. This can easily be done by the following command:. Once again, we can show this using an example. We will be downloading two HTML files from two different websites.

For better understanding, please look at the image below:. Here filename refers to the name that you want to address the file as. Using this, we can also change the type of the file. This is shown in the image below:. Wget also allows users to recursively download their files which is basically downloading all the files from the website under a single directory. For more information regarding Wget, users can input the following command into the terminal to get access to all the Wget commands that appear to be available:.

Curl is another command line tool that can be used to download files from the internet. Unlike Wget, which is command line only, features of Curl are powered by libcurl which is a cross-platform URL transfer library. Curl not only allows downloading of files but can also be used for uploading and exchanging of requests with servers.

To download a binary file, we can use the -O parameter which pulls down the content exactly as the source file specified dictates, including the name as such:. We can achieve the same result by using the -o parameter option also and specifying a target filename:. This is handy if you want to change the name of the file on the target filesystem. You can pass along all sorts of variable goodness into the name for output when you want to do something programmatically, which is all sorts of awesome as you get to using cURL for automated file management and other neat functions.

Sign me up for the newsletter! To do this, we add the -I option to the curl command as follows. Here is a quick example to show how the second way works in bash scripts that can be used to serve as a part of a web page health checker.

Often, we want to get the response code for a curl request in bash. To do this, we would need to first request the headers of a response and then extract the response code.

Here is what it would look like. There will come a time when you need to make posts with curl in bash to authenticate to access or modification of private content. Such is the case working with APIs and html forms. It may require multiple curl requests. The placeholder curl command line for this way is as follows. Making posts involves adding corresponding headers and data to allow for authentication.

Here is an example of using curl in bash scripts to download a file requiring basic authentication. Note that credentials are stored in a separate file called bash-curl-basic-auth-example-config. Source: curl-basic-auth. Source: curl-basic-auth-config. Here you see how writing a bash script allows you to avoid having to include your secrets in the command line. The magic of bash is that you can do just about anything you have an intent to do.

Jumping through the hoops of csrf protection is one way to kill it with curl in bash scripts. In modern web applications there is a security feature called csrf protection to prevent posts requests from anywhere without established access to the site in question.

Here what your bash script may look like to gain authorized access to a page content with csrf protection. Source: curl-example. Notes on script It uses a alias called commands that I mentioned in a previous post about the bash declare command , which makes it possible to declare subcommands implicitly by way of convention. Here you see that bash can be used to string curl request together with logic to carry out the intent of your script.

Source: subshell-functions. Here is curl request command line in a bash script that I wrote in late back before switching over to Trello. Source: wonderlist. If you are not familiar with the option, it is set the request command to use.