[Solved] CURLOPT_FOLLOWLOCATION not working correctly on Dreamhost? - Printable Version
-Shoutbox (https://shoutbox.menthix.net)
+-- Forum: MsgHelp Archive (/forumdisplay.php?fid=58)
+--- Forum: Skype & Technology (/forumdisplay.php?fid=9)
+---- Forum: Tech Talk (/forumdisplay.php?fid=17)
+----- Thread: [Solved] CURLOPT_FOLLOWLOCATION not working correctly on Dreamhost? (/showthread.php?tid=92856)
[Solved] CURLOPT_FOLLOWLOCATION not working correctly on Dreamhost? by Mike on 11-08-2009 at 07:01 AM
Hey, I have the following code which gets returns the headers of a website:
php code:
function head($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 7);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1");
curl_setopt($ch, CURLOPT_NOBODY, true);
$output = curl_exec($ch);
// check for errors
if (curl_errno($ch))
{
echo curl_error($ch);
$output = false;
}
curl_close($ch);
return $output;
}
This works perfectly on my local server, but when I upload it to Dreamhost, it only returns the headers for the first website, and does not follow the location header (EDIT: the first headers are only shown if I add a [font=courier]echo $output; after the curl_exec line). It also throws this error:
quote:
Originally posted by curl
transfer closed with outstanding read data remaining
Does anyone know what's going on?
Thanks
Sorry for the lack of information. I'm in a hurry now!
RE: CURLOPT_FOLLOWLOCATION not working correctly on Dreamhost? by Vilkku on 11-08-2009 at 09:59 AM
Did you check their WIKI? Don't know if it is of any use for you.
RE: CURLOPT_FOLLOWLOCATION not working correctly on Dreamhost? by Mike on 11-08-2009 at 01:33 PM
quote:Yes, and it didn't help
Originally posted by Vilkku
Did you check their WIKI? Don't know if it is of any use for you.
Could it be because of the chunked transfer-encoding? Because it works for a website that doesn't use chunked transfer-encoding...
By the way, the server uses libcurl/7.13.2, while my local server uses libcurl/7.16.0. Maybe it's a bug with older curl versions?
RE: CURLOPT_FOLLOWLOCATION not working correctly on Dreamhost? by WDZ on 11-08-2009 at 10:40 PM
quote:wtf, seriously? That version is over 4 years old... :/
Originally posted by Mike
By the way, the server uses libcurl/7.13.2
The only thing I can suggest is to enable verbose mode, which will show more detailed information...
php code:(Look into CURLOPT_STDERR if you want to redirect the info to a file)
curl_setopt($ch, CURLOPT_VERBOSE, true);
My guess is that cURL is misinterpreting the response somehow, thinking it got a partial response when in fact it got the full response.
RE: CURLOPT_FOLLOWLOCATION not working correctly on Dreamhost? by Mike on 11-09-2009 at 12:40 AM
quote:Don't blame me; blame DreamHost
Originally posted by WDZ
wtf, seriously? That version is over 4 years old... :/
quote:Here's what verbose mode says...
Originally posted by WDZ
The only thing I can suggest is to enable verbose mode, which will show more detailed information...
Localhost:
quote:
Originally posted by Local verbose curl
* About to connect() to www.youtube.com port 80 (#0)
* Trying 74.125.19.100... * connected
* Connected to www.youtube.com (74.125.19.100) port 80 (#0)
> HEAD /get_video?video_id=S6fuwodARgc&t=vjVQa1PpcFNh9Oe1U09B1yeF1j7YskSZbCSI0L-dF5A%3D HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
Host: www.youtube.com
Accept: */*
< HTTP/1.1 303 See Other
< Date: Mon, 09 Nov 2009 00:30:14 GMT
< Server: Apache
< X-Content-Type-Options: nosniff
< Set-Cookie: use_hitbox=72c46ff6cbcdc4c5585c36411b6b334edAEAAAAw; path=/; domain=.youtube.com
< Set-Cookie: VISITOR_INFO1_LIVE=OrEeGk9ygUk; path=/; domain=.youtube.com; expires=Wed, 07-Jul-2010 00:30:14 GMT
< Set-Cookie: GEO=c3dfff1fcbd3b12c430d1165a81c74c5cwsAAAAzR1JeROxvSvdilg==; path=/; domain=.youtube.com
< Expires: Tue, 27 Apr 1971 19:44:06 EST
< X-YouTube-MID: WkFSZzctYUFHdmd1X0NrNXRMYm5CbnRHdFM4dWlDdE1WSE5jNWJXeEp4V1ZGOEtMOWtTR0J3
< Cache-Control: no-cache
< Location: http://v3.lscache4.c.youtube.com/videoplayback?ip...d=4ba7eec287404607
< Content-Type: text/html; charset=utf-8
< X-XSS-Protection: 0
< Transfer-Encoding: chunked
* Issue another request to this URL: 'http://v3.lscache4.c.youtube.com/videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=901301&algorithm=throttle-factor&itag=5&ipbits=0&burst=40&sver=3&expire=1257750000&key=yt1&signature=72C6EE4AB70ED328CBF5E6AC6E23F7291C60721B.12C0C1E286ED25BB9BF5B6E801CA0868F42BB319&factor=1.25&id=4ba7eec287404607'
* Disables POST, goes with HEAD
* Examining connection #0 for reuse
* About to connect() to v3.lscache4.c.youtube.com port 80 (#1)
* Trying 74.125.105.22... * connected
* Connected to v3.lscache4.c.youtube.com (74.125.105.22) port 80 (#1)
> HEAD /videoplayback?ip=0.0.0.0&sparams=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor&fexp=901301&algorithm=throttle-factor&itag=5&ipbits=0&burst=40&sver=3&expire=1257750000&key=yt1&signature=72C6EE4AB70ED328CBF5E6AC6E23F7291C60721B.12C0C1E286ED25BB9BF5B6E801CA0868F42BB319&factor=1.25&id=4ba7eec287404607 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
Host: v3.lscache4.c.youtube.com
Accept: */*
< HTTP/1.1 200 OK
< Last-Modified: Sat, 16 May 2009 20:46:13 GMT
< Content-Type: video/x-flv
< Content-Length: 7358674
< Connection: close
< Content-Disposition: attachment; filename="video.flv"
< Expires: Mon, 09 Nov 2009 01:30:14 GMT
< Cache-Control: public,max-age=3600
< Date: Mon, 09 Nov 2009 00:30:14 GMT
< Server: gvs 1.0
* Closing connection #0
* Closing connection #1
Dreamhost:
quote:
Originally posted by Dreamhost's verbose curl
* About to connect() to www.youtube.com port 80
* Trying 74.125.19.100... * connected
* Connected to www.youtube.com (74.125.19.100) port 80
> HEAD /get_video?video_id=S6fuwodARgc&t=vjVQa1PpcFPBrvsyIEO7JPQnqs44Gr18QBSR34qJxdY%3D HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
Host: www.youtube.com
Pragma: no-cache
Accept: */*
< HTTP/1.1 303 See Other
< Date: Mon, 09 Nov 2009 00:29:40 GMT
< Server: Apache
< X-Content-Type-Options: nosniff
< Set-Cookie: use_hitbox=72c46ff6bbcbb7c5585c36411b6b334edAEAAAAw; path=/; domain=.youtube.com
< Set-Cookie: VISITOR_INFO1_LIVE=4pDVqtCEKd0; path=/; domain=.youtube.com; expires=Wed, 07-Jul-2010 00:29:40 GMT
< Set-Cookie: GEO=b5b70b357bdc62dd1a2592d3770a1bd8cwsAAAAzVVPQcfHzSvdidA==; path=/; domain=.youtube.com
< Expires: Tue, 27 Apr 1971 19:44:06 EST
< X-YouTube-MID: WkFSZzctYUeddmdxOXBtV1ZsdjVJR0NiMGdoV2lLdU9tUG1CQ3lDYmRfS3BOLUlSTGp5c2J3
< Cache-Control: no-cache
< Location: http://v3.lscache4.c.youtube.com/videoplayback?ip...d=4ba7eec287404607
< Content-Type: text/html; charset=utf-8
< X-XSS-Protection: 0
< Transfer-Encoding: chunked
* transfer closed with outstanding read data remaining
* Closing connection #0
As you can see, it pretty much says the same things as before...
Note: Visiting the first URL directly probably won't work. You'll have to figure out the "t" parameter yourself.
RE: CURLOPT_FOLLOWLOCATION not working correctly on Dreamhost? by WDZ on 11-09-2009 at 03:13 AM
I'm pretty sure that's actually a bug that was fixed in cURL 7.14.0.
You might be able to work around it by telling the remote server not to send you a chunked response, but the only way I can think of to do that is to downgrade to HTTP 1.0.
php code:
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_0);
RE: CURLOPT_FOLLOWLOCATION not working correctly on Dreamhost? by Mike on 11-09-2009 at 05:53 AM
Thanks WDZ, it's working
What are the major disadvantages by using HTTP 1.0 though?
RE: [Solved] CURLOPT_FOLLOWLOCATION not working correctly on Dreamhost? by WDZ on 11-09-2009 at 06:22 AM
quote:I dunno, maybe there are none... depends on the application. Tons of people use wget, which still uses HTTP/1.0 for all its requests.
Originally posted by Mike
What are the major disadvantages by using HTTP 1.0 though?