Simon Willison’s Weblog

Grabbing web pages with Perl and PHP

1st September 2002

Web Basics with LWP (via Scott) is an excellent tutorial on Perl’s LWP, a powerful set of modules which make it easy to retrieve content from the web. I’ve been using the excellent Snoopy class for PHP for the same purpose, but I have to admit it isn’t half as comprehensive as LWP. I’ve also written my own simple function safeGet for more light weight tasks—it grabs and returns the contents of a web page but limits both the size of the page and the maximum time it can take to download it.