Shoutbox

Hey, I have the following code which gets returns the headers of a website:

PHP code:

---

function head($url)
{
    $ch = curl_init();
 
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_HEADER, true);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_MAXREDIRS, 7);
    curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1");
    curl_setopt($ch, CURLOPT_NOBODY, true);
 
   
    $output = curl_exec($ch);
    // check for errors
    if (curl_errno($ch))
    {
        echo curl_error($ch);
        $output = false;
    }
    curl_close($ch);
    return $output;
}

---

This works perfectly on my local server, but when I upload it to Dreamhost, it only returns the headers for the first website, and does not follow the location header (EDIT: the first headers are only shown if I add a [font=courier]echo $output; after the curl_exec line). It also throws this error:

quote:

---

Originally posted by curl
transfer closed with outstanding read data remaining

---

Does anyone know what's going on?

Thanks

Sorry for the lack of information. I'm in a hurry now!

Did you check their WIKI? Don't know if it is of any use for you.

quote:

---

Originally posted by Vilkku
Did you check their WIKI? Don't know if it is of any use for you.

---

Yes, and it didn't help

Could it be because of the chunked transfer-encoding? Because it works for a website that doesn't use chunked transfer-encoding...

By the way, the server uses libcurl/7.13.2, while my local server uses libcurl/7.16.0. Maybe it's a bug with older curl versions?

quote:

---

Originally posted by Mike
By the way, the server uses libcurl/7.13.2

---

wtf, seriously? That version is over 4 years old... :/

The only thing I can suggest is to enable verbose mode, which will show more detailed information...

PHP code:

---

curl_setopt($ch, CURLOPT_VERBOSE, true);

---

(Look into CURLOPT_STDERR if you want to redirect the info to a file)

My guess is that cURL is misinterpreting the response somehow, thinking it got a partial response when in fact it got the full response.

quote:

---

Originally posted by WDZ
wtf, seriously? That version is over 4 years old... :/

---

Don't blame me; blame DreamHost

quote:

---

Originally posted by WDZ
The only thing I can suggest is to enable verbose mode, which will show more detailed information...

---

Here's what verbose mode says...

Localhost:

quote:

---

Originally posted by Local verbose curl
* About to connect() to www.youtube.com port 80 (#0)
*   Trying 74.125.19.100... * connected
* Connected to www.youtube.com (74.125.19.100) port 80 (#0)
> HEAD /get_video?video_id=S6fuwodARgc&t=vjVQa1PpcFNh9Oe1U09B1yeF1j7YskSZbCSI0L-dF5A%3D HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
Host: www.youtube.com
Accept: */*

< HTTP/1.1 303 See Other
< Date: Mon, 09 Nov 2009 00:30:14 GMT
< Server: Apache
< X-Content-Type-Options: nosniff
< Set-Cookie: use_hitbox=72c46ff6cbcdc4c5585c36411b6b334edAEAAAAw; path=/; domain=.youtube.com
< Set-Cookie: VISITOR_INFO1_LIVE=OrEeGk9ygUk; path=/; domain=.youtube.com; expires=Wed, 07-Jul-2010 00:30:14 GMT
< Set-Cookie: GEO=c3dfff1fcbd3b12c430d1165a81c74c5cwsAAAAzR1JeROxvSvdilg==; path=/; domain=.youtube.com
< Expires: Tue, 27 Apr 1971 19:44:06 EST
< X-YouTube-MID: WkFSZzctYUFHdmd1X0NrNXRMYm5CbnRHdFM4dWlDdE1WSE5jNWJXeEp4V1ZGOEtMOWtTR0J3
< Cache-Control: no-cache
< Location: http://v3.lscache4.c.youtube.com/videoplayback?ip...d=4ba7eec287404607
< Content-Type: text/html; charset=utf-8
< X-XSS-Protection: 0
< Transfer-Encoding: chunked
* Issue another request to this URL: 'http://v3.lscache4.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=901301&algorithm=throttle-factor&itag=5&ipbits=0&burst=40&sver=3&expire=1257750000&key=yt1&signature=72C6EE4AB70ED328CBF5E6AC6E23F7291C60721B.12C0C1E286ED25BB9BF5B6E801CA0868F42BB319&factor=1.25&id=4ba7eec287404607'
* Disables POST, goes with HEAD
* Examining connection #0 for reuse
* About to connect() to v3.lscache4.c.youtube.com port 80 (#1)
*   Trying 74.125.105.22... * connected
* Connected to v3.lscache4.c.youtube.com (74.125.105.22) port 80 (#1)
> HEAD /videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=901301&algorithm=throttle-factor&itag=5&ipbits=0&burst=40&sver=3&expire=1257750000&key=yt1&signature=72C6EE4AB70ED328CBF5E6AC6E23F7291C60721B.12C0C1E286ED25BB9BF5B6E801CA0868F42BB319&factor=1.25&id=4ba7eec287404607 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
Host: v3.lscache4.c.youtube.com
Accept: */*

< HTTP/1.1 200 OK
< Last-Modified: Sat, 16 May 2009 20:46:13 GMT
< Content-Type: video/x-flv
< Content-Length: 7358674
< Connection: close
< Content-Disposition: attachment; filename="video.flv"
< Expires: Mon, 09 Nov 2009 01:30:14 GMT
< Cache-Control: public,max-age=3600
< Date: Mon, 09 Nov 2009 00:30:14 GMT
< Server: gvs 1.0
* Closing connection #0
* Closing connection #1

---

Dreamhost:

quote:

---

Originally posted by Dreamhost's verbose curl
* About to connect() to www.youtube.com port 80
*   Trying 74.125.19.100... * connected
* Connected to www.youtube.com (74.125.19.100) port 80
> HEAD /get_video?video_id=S6fuwodARgc&t=vjVQa1PpcFPBrvsyIEO7JPQnqs44Gr18QBSR34qJxdY%3D HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
Host: www.youtube.com
Pragma: no-cache
Accept: */*

< HTTP/1.1 303 See Other
< Date: Mon, 09 Nov 2009 00:29:40 GMT
< Server: Apache
< X-Content-Type-Options: nosniff
< Set-Cookie: use_hitbox=72c46ff6bbcbb7c5585c36411b6b334edAEAAAAw; path=/; domain=.youtube.com
< Set-Cookie: VISITOR_INFO1_LIVE=4pDVqtCEKd0; path=/; domain=.youtube.com; expires=Wed, 07-Jul-2010 00:29:40 GMT
< Set-Cookie: GEO=b5b70b357bdc62dd1a2592d3770a1bd8cwsAAAAzVVPQcfHzSvdidA==; path=/; domain=.youtube.com
< Expires: Tue, 27 Apr 1971 19:44:06 EST
< X-YouTube-MID: WkFSZzctYUeddmdxOXBtV1ZsdjVJR0NiMGdoV2lLdU9tUG1CQ3lDYmRfS3BOLUlSTGp5c2J3
< Cache-Control: no-cache
< Location: http://v3.lscache4.c.youtube.com/videoplayback?ip...d=4ba7eec287404607
< Content-Type: text/html; charset=utf-8
< X-XSS-Protection: 0
< Transfer-Encoding: chunked
* transfer closed with outstanding read data remaining
* Closing connection #0

---

As you can see, it pretty much says the same things as before...

Note: Visiting the first URL directly probably won't work. You'll have to figure out the "t" parameter yourself.

I'm pretty sure that's actually a bug that was fixed in cURL 7.14.0.

You might be able to work around it by telling the remote server not to send you a chunked response, but the only way I can think of to do that is to downgrade to HTTP 1.0.

PHP code:

---

curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_0);

---

Thanks WDZ, it's working

What are the major disadvantages by using HTTP 1.0 though?

quote:

---

Originally posted by Mike
What are the major disadvantages by using HTTP 1.0 though?

---

I dunno, maybe there are none... depends on the application. Tons of people use wget, which still uses HTTP/1.0 for all its requests.