Weblog entry #7 for endecotp
See all weblog entries made by endecotp.
#7
httping, wget, curl: none does what I want!
Posted by endecotp on Tue 14 Apr 2009 at 20:10
I was expecting that this would be easy, but I haven't yet found the right tool for the job. I have a long list of URLs, and for each one I want to do an HTTP HEAD request to retrieve the headers. If the initial request gets a redirect response I want to follow the redirect so that I end up with probably either a 200 OK or a 404 not found, and I want to log the content-type header in that final response. It's important that I do a HEAD request not a GET because these are large files, and I have a lot of them to look at.
I was expecting that Wget would have an option to send a HEAD, but it doesn't seem to. Neither does Curl. httping sends a HEAD, but it doesn't follow the redirect.
Can anyone point me in the direction of a simple utility that does this, or shall I write my own?
I was expecting that Wget would have an option to send a HEAD, but it doesn't seem to. Neither does Curl. httping sends a HEAD, but it doesn't follow the redirect.
Can anyone point me in the direction of a simple utility that does this, or shall I write my own?
Comments on this Entry
Posted by Steve (2001:0xx:0xx:0xxx:0xxx:0xxx:xx) on Tue 14 Apr 2009 at 20:50
[ Send Message | View Steve's Scratchpad | View Weblogs ]
[ Send Message | View Steve's Scratchpad | View Weblogs ]
skx@gold:~$ HEAD www.debian-administration.org 200 OK Connection: close Date: Tue, 14 Apr 2009 19:49:21 GMT Server: Apache/2.2.9 Vary: Accept-Encoding Content-Type: text/html Client-Date: Tue, 14 Apr 2009 19:49:21 GMT Client-Peer: 89.16.161.98:80 Client-Response-Num: 1 Set-Cookie: CGISESSID=0ddfeed40pies381656665f1928ea75e49d; path=/; expires=Tue, 21-Apr-2009 19:49:21 GMT; HttpOnly X-Pad: avoid browser bug
The HEAD command comes from the libwww-perl package.
[ Parent | Reply to this comment ]
Thanks Steve! I've checked and yes it does seem to follow redirects. Great!
[ Parent | Reply to this comment ]