Curl is a utility that allows you to transfer data to or from a server using many different protocols. It is most commonly use to fetch web pages or information from the web to the Linux command line. In this Linux quick tip we will show you how to grep the output of the curl command. This comes in especially handy when you are using verbose mode to capture data like cookies, protocols, and header information.
Jump directly to "How to grep curl output"
The Problem with Using Grep on Curl Output
If you attempt to pipe the output of the curl command to grep you will get an unexpected result. Basically, the grep command will be ignored and you will see the full output of curl. The reason for this is curl sends all of it's output to standard error (STDERR). Most command line utilities use standard output (STDOUT).
When you use a pipe
'|' it passes standard output (STDOUT) of the command on the left, to the command on the right. Since curl output is standard error (STDERR), it is not passed through the pipe, rather printed to the terminal.
If you are not familiar with the standard streams (STDIN, STDOUT and STDERR) read "Linux IO, Standard Streams, and Redirection".
Let's look at two methods for capturing the curl output and piping it to grep.
Redirect Curl Output to STDOUT to Allow Grep
The most common way of using grep with curl is to redirect STDERR to STDOUT. Once you redirect the standard error stream to standard output you can use grep as you normally would.
[mcherisi@putor ~]$ curl -v https://www.putorius.net/grep-string-from-a-variable.html 2>&1 | grep cache-control < cache-control: max-age=31536000, must-revalidate
Using Curl Options to Allow Grep
The --stderr option in curl allows you to redirect the standard error (STDERR) to a file. If you specify the filename as
- (a single dash) the output will be written to standard output. This allows you to pipe it to grep without any shell redirection.
[mcherisi@putor ~]$ curl -v --stderr - https://www.putorius.net/grep-string-from-a-variable.html | grep cache-control < cache-control: max-age=31536000, must-revalidate
Once you understand standard streams and redirection this workaround will seem natural. It also helps to know that curl used standard error (STDERR) for all of it's output. This is not normal behavior for a command line utility.