HTTP Request Diagnostics With cURL

I had recently needed to diagnose some HTTP requests to understand where time was spent, download speeds, etc. This great post by Joseph Scott shows how to get timing info for a request using curl. I expanded on it by adding all available curl --write-out variables and HTTP headers (sample response below).

Here’s what you need to do:

Create a file for the curl formatting (named curl-format in our case).
Add the following to curl-format:

\n\n
      content_type: %{content_type}\n
filename_effective: %{filename_effective}\n
    ftp_entry_path: %{ftp_entry_path}\n
         http_code: %{http_code}\n
      http_connect: %{http_connect}\n
          local_ip: %{local_ip}\n
        local_port: %{local_port}\n
      num_connects: %{num_connects}\n
     num_redirects: %{num_redirects}\n
      redirect_url: %{redirect_url}\n
         remote_ip: %{remote_ip}\n
       remote_port: %{remote_port}\n
     size_download: %{size_download}\n
       size_header: %{size_header}\n
      size_request: %{size_request}\n
       size_upload: %{size_upload}\n
    speed_download: %{speed_download}\n
      speed_upload: %{speed_upload}\n
 ssl_verify_result: %{ssl_verify_result}\n
     url_effective: %{url_effective}\n
\n\n
   time_namelookup: %{time_namelookup}\n
      time_connect: %{time_connect}\n
   time_appconnect: %{time_appconnect}\n
  time_pretransfer: %{time_pretransfer}\n
     time_redirect: %{time_redirect}\n
time_starttransfer: %{time_starttransfer}\n
                   -------\n
        time_total: %{time_total}\n
\n

Then make a request with the format file:

$ curl -v -w "@curl-format" -o /dev/null -s http://www.orrsella.com

Example response:

* Adding handle: conn: 0x7f8b3b803600
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7f8b3b803600) send_pipe: 1, recv_pipe: 0
* About to connect() to www.orrsella.com port 80 (#0)
*   Trying 104.131.121.209...
* Connected to www.orrsella.com (104.131.121.209) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.30.0
> Host: www.orrsella.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
* Server nginx/1.4.6 (Ubuntu) is not blacklisted
< Server: nginx/1.4.6 (Ubuntu)
< Date: Mon, 06 Oct 2014 13:42:42 GMT
< Content-Type: text/html
< Content-Length: 193
< Connection: keep-alive
< Location: http://orrsella.com/
<
{ [data not shown]
* Connection #0 to host www.orrsella.com left intact


      content_type: text/html
filename_effective: /dev/null
    ftp_entry_path:
         http_code: 301
      http_connect: 000
          local_ip: 192.168.28.249
        local_port: 58388
      num_connects: 1
     num_redirects: 0
      redirect_url: http://orrsella.com/
         remote_ip: 104.131.121.209
       remote_port: 80
     size_download: 193
       size_header: 203
      size_request: 80
       size_upload: 0
    speed_download: 689.000
      speed_upload: 0.000
 ssl_verify_result: 0
     url_effective: http://www.orrsella.com/


   time_namelookup: 0.001
      time_connect: 0.141
   time_appconnect: 0.000
  time_pretransfer: 0.141
     time_redirect: 0.000
time_starttransfer: 0.280
                   -------
        time_total: 0.280

Check out the curl man page for a detailed explanation of each of the used %{variable}s.