Comments on: Scraping article directories by exploiting search functionality

By: Peter

Peter — Fri, 29 Jan 2010 13:46:56 +0000

Try setting up and using TOR (http://www.torproject.org/) with your scrapers instead of searching for public proxies that are actually working.

Its slow but works fine for scraping google/yahoo etc.

By: Briefcases

Briefcases — Tue, 03 Nov 2009 00:29:18 +0000

Wow, even combining and/or rewriting scraped articles is also a great idea. Not only can you scrape a ton of content, but you can even make the search engines think it is all unique.

By: seozero

seozero — Thu, 29 Oct 2009 20:34:38 +0000

@serial number, orly?

By: seozero

seozero — Thu, 29 Oct 2009 20:27:32 +0000

@Winalot, no. My mistake :}

By: serial number

serial number — Thu, 29 Oct 2009 01:43:35 +0000

@serozero, You can get full list of urls also from search engine. Ex : site:ezinearticles.com .

By: Winalot

Winalot — Wed, 28 Oct 2009 16:10:24 +0000

OT: So are you scraping and storing Geocities pages now before they get de-indexed?

By: seozero

seozero — Wed, 28 Oct 2009 07:33:02 +0000

@serial number, sitemap scraping is another alternative. But what I like about search exploiting is that I get only keyword related articles.

By: seozero

seozero — Wed, 28 Oct 2009 07:29:37 +0000

@winalot, depends on website. I rewrite/spin/outsource only if I need some quality and uniqueness.

By: Winalot

Winalot — Tue, 27 Oct 2009 21:16:49 +0000

Do you re-write/combine etc. the articles once scraped?

By: Metal Briefcases

Metal Briefcases — Tue, 27 Oct 2009 20:47:12 +0000

Wow, you are opening my eyes to new opportunities. I definitely have to get on board and get some article scraping going. This could be very powerful.