Comments on: Scraping websites with PHP cURL under proxy

By: root

root — Sat, 17 Dec 2011 19:17:35 +0000

When using proxies (in my case for a proxy tester) my script just keeps running as soon as i have more than 5 curls in a loop, even with the multiple curl handler … cant figure out why that is?!

By: Scraper

Scraper — Thu, 15 Dec 2011 18:35:18 +0000

That’s good information in this blog, thanks for sharing.
I’m currently using the Google scraper on google-scraper.squabbel.com (a bit customized). I think I found that link actually on one of your blogs a while ago!

I need to scrape about 1 million keywords per month with it, do you know how many proxies I will need ?
I’m using a “guru” license at seo-proxies at this time, not sure if it’s enough to keep going or if I need to upgrade even higher.

This is an important project, that’s why I’d really appreciate an answer.

By: Status

Status — Wed, 02 Nov 2011 21:30:35 +0000

Hey, I´ve tested it that way with many different proxies, but I always get “Received HTTP code 403 from proxy after CONNECT” error. Any ideas what to do?

By: Champ Polestico

Champ Polestico — Fri, 28 Oct 2011 04:50:29 +0000

Thanks man! Saved my life.

By: Hanibal

Hanibal — Wed, 12 Oct 2011 20:23:23 +0000

Hello.
Thanks for post.
I have problem with running that code.
Error 404.
I have proxy from http://www.xroxy.com/proxy-country-DE.htm and it work when i put it into settings in firefox.
Why it don’t work when I’m doing it like this: ?
$options = array(
CURLOPT_FAILONERROR => True,
CURLOPT_RETURNTRANSFER => True,
CURLOPT_TIMEOUT => 25,
CURLOPT_SSL_VERIFYPEER => False,
CURLOPT_USERAGENT => ‘Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)’,
CURLOPT_FOLLOWLOCATION => True,
CURLOPT_POST => 0,
CURLOPT_COOKIEJAR => $cookiesFileName,
CURLOPT_COOKIEFILE => $cookiesFileName,
CURLOPT_PROXY => ’62.157.186.16:80′,
CURLOPT_HTTPPROXYTUNNEL => 1,
);
$URL= ‘http://hanibal.hopto.org/';
$c = curl_init($URL);
curl_setopt_array($c, $options);
$result = curl_exec($c);

By: Ejz

Ejz — Thu, 29 Sep 2011 14:07:39 +0000

Can you provide some examples concerning multithread issue?

By: Google Scraping

Google Scraping — Wed, 24 Aug 2011 00:23:33 +0000

Yeah, I’ve used this post several times to help clients use our proxies, and it seems to work out well.

By: Aro

Aro — Mon, 15 Aug 2011 15:28:28 +0000

Theres a python libray called libgmail that lets you do that.

By: mrmoney

mrmoney — Mon, 24 Jan 2011 16:48:19 +0000

how can i do this to search emails on google ?

By: vijay vinoth

vijay vinoth — Mon, 27 Sep 2010 21:12:07 +0000

Hi all,

I tried this coading. But I got the error message like this. Any one can help me ???

Warning: curl_setopt() [function.curl-setopt]: CURLOPT_FOLLOWLOCATION cannot be activated when in safe_mode or an open_basedir is set in

/home/wwwlawy/public_html/mails/gmail/libgmailer.php on line 42

Warning: curl_setopt() [function.curl-setopt]: CURLOPT_FOLLOWLOCATION cannot be activated when in safe_mode or an open_basedir is set in /home/wwwlawy/public_html/mails/gmail/libgmailer.php on line 77

No contacts found