Yesterday I ran into a tough problem at work, (i.e. Google didn’t return a web page describing the solution). I spent almost a whole day trying to figure this out, and the solution turned out to be incredibly easy.
We are building a new web server ([PHP][], [Apache][], [FreeBSD][]) under pretty strict security guidelines. In particular, this server isn’t allowed to do outbound http[s] queries except through a proxy. You may be asking why a web server needs to issue http queries itself — in our case, we’re looking up shipping prices at UPS, FedEx, etc. using [PHP’s cURL module][PHP_cURL]. When we finally got around to testing our PHP script, it didn’t fail, but it didn’t actually return any results either.
Our script didn’t have much in the way of error checking, in particular on the [curl_setopt][] calls.
I tried adding some diagnostics, hoping to get some helpful error messages:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.ups.com') or die(curl_error($ch));
echo "url set\n";
curl_setopt($ch, CURLOPT_HEADER, TRUE) or die(curl_error($ch));
echo "header set\n";
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE) or die(curl_error($ch));
echo "returntransfer set\n";
curl_setopt($ch, CURLOPT_PROXY, 'http://proxy.example.com:3128') or die(curl_error($ch));
echo "proxy set\n";
$data = curl_exec($ch);
echo 'Results: ', $data;
echo "\n";
curl_close($ch);
The only thing we found from this script was that the final curl\_setopt (for CURLOPT\_PROXY) was failing, but there was no output from curl_error.
I chased after a few red herrings at this point:
* One suggestion (I believe it was on the [curl_setopt][] user-contributed documentation) suggested that Apache needed to have mod\_proxy and mod\_proxy\_http loaded. This didn’t solve the problem for us, and we removed these (after finding the real solution) to verify that they weren’t needed.
* Another post suggested that we turn off CURLOPT\_HTTPPROXYTUNNEL. As far I as know, this one didn’t make any difference for our app.
* The final red herring suggested that we set CURLOPT\_HTTPHEADER since we were requesting an XML document. It also suggested that we disable caching with CURLOPT\_FORBID\_REUSE and CURLOPT\_FRESH\_CONNECT. These turned out to be unnecessary for us.
At this point I was running out of ideas. I finally decided to try the [cURL][] utility itself from the command line:
curl --proxy http://proxy.example.com:3128 --verbose http://www.ups.com/
Unfortunately I closed the terminal window with the output of this command, but if I recall correctly it showed that the proxy wasn’t being queried and curl was failing over to (attempt) directly accessing the target. Some more digging and I discovered that curl had been built without proxy support. As soon as I rebuilt ftp/curl WITH_PROXY=TRUE the curl command worked flawlessly, as did our PHP script.
[PHP]: http://php.net/ “PHP scripting language”
[Apache]: http://www.apache.org/ “Apache web server”
[FreeBSD]: http://www.freebsd.org/ “FreeBSD Unix”
[PHP_cURL]: http://www.php.net/manual/en/curl.examples-basic.php “Client URL library for PHP”
[curl_setopt]: http://www.php.net/manual/en/function.curl-setopt.php “Documentation for PHP’s curl_setopt function”
[cURL]: http://curl.haxx.se/ “The cURL library and CLI tool”
