You must login or register!
Inbound.org uses Twitter to register and create accounts. Your Twitter handle will also be your username here on Inbound and registration/login will enable you to submit content, post comments and create/edit your Inbound profile. Use the button below to verify your Twitter account.
Login or Register
on Dec 10 2012 3 comments3 DISCUSSING
Great article on the art of scraping. The one issue it failed to bring up was depending on the content you are scraping, you could run into some duplicate content problems with Google.
It depends on what you actually *do* with the content. I left it up to the imagination of the scraper to decide how to use those methods. Obviously, if you're re-posting other site's content as-is, you're probably going to have some issues (not just SEO ones) especially if the site who got scraped feels like you're damaging their branding or hurting their revenue stream.
If you're using python as alluded to in the post, I'd suggest taking a look at Scrapy http://scrapy.org/. It's got a great parser, and a bunch of other built in niceties like the ability to write middlewares so you can have a full crawler, and not have to cobble other libraries together to get at what you need.