<p>What built-in PHP functions are useful for web scraping?  What are some good resources (web or print) for getting up to speed on web scraping with PHP?</p>

[{'answer_id': 26951, 'author': 'Peter Stuifzand', 'author_id': 1633, 'author_profile': 'https://Stackoverflow.com/users/1633', 'pm_score': -1, 'selected': False, 'text': '<p>The curl library allows you to download web pages. You should look into regular expressions for doing the scraping.</p>\n'}, {'answer_id': 26955, 'author': 'Brian Warshaw', 'author_id': 1344, 'author_profile': 'https://Stackoverflow.com/users/1344', 'pm_score': 0, 'selected': False, 'text': '<p><code>file_get_contents()</code> can take a remote URL and give you the source. You can then use regular expressions (with the Perl-compatible functions) to grab what you need.</p>\n\n<p>Out of curiosity, what are you trying to scrape?</p>\n'}, {'answer_id': 26972, 'author': 'dlamblin', 'author_id': 459, 'author_profile': 'https://Stackoverflow.com/users/459', 'pm_score': 0, 'selected': False, 'text': "<p>I'd either use libcurl or Perl's LWP (libwww for perl). Is there a libwww for php?</p>\n"}, {'answer_id': 103554, 'author': 'tyshock', 'author_id': 16448, 'author_profile': 'https://Stackoverflow.com/users/16448', 'pm_score': 6, 'selected': False, 'text': '<p>Scraping generally encompasses 3 steps: </p>\n\n<ul>\n<li>first you GET or POST your request\nto a specified URL </li>\n<li>next you receive\n    the html that is returned as the\n    response</li>\n<li>finally you parse out of\n    that html the text you\'d like to\n    scrape.</li>\n</ul>\n\n<p>To accomplish steps 1 and 2, below is a simple php class which uses Curl to fetch webpages using either GET or POST.  After you get the HTML back, you just use Regular Expressions to accomplish step 3 by parsing out the text you\'d like to scrape.</p>\n\n<p>For regular expressions, my favorite tutorial site is the following:\n<a href="http://www.regular-expressions.info/" rel="nofollow noreferrer">Regular Expressions Tutorial</a></p>\n\n<p>My Favorite program for working with RegExs is <a href="http://www.regexbuddy.com/" rel="nofollow noreferrer">Regex Buddy</a>.  I would advise you to try the demo of that product even if you have no intention of buying it.  It is an invaluable tool and will even generate code for your regexs you make in your language of choice (including php).</p>\n\n<p>Usage:\n<pre><code></p>\n\n<p>$curl = new Curl();\n$html = $curl->get("<a href="http://www.google.com" rel="nofollow noreferrer">http://www.google.com</a>");</p>\n\n<p>// now, do your regex work against $html\n</pre></code></p>\n\n<p>PHP Class:</p>\n\n<pre><code>\n\n&lt;?php\n\nclass Curl\n{       \n\n    public $cookieJar = "";\n\n    public function __construct($cookieJarFile = \'cookies.txt\') {\n        $this->cookieJar = $cookieJarFile;\n    }\n\n    function setup()\n    {\n\n\n        $header = array();\n        $header[0] = "Accept: text/xml,application/xml,application/xhtml+xml,";\n        $header[0] .= "text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";\n        $header[] =  "Cache-Control: max-age=0";\n        $header[] =  "Connection: keep-alive";\n        $header[] = "Keep-Alive: 300";\n        $header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";\n        $header[] = "Accept-Language: en-us,en;q=0.5";\n        $header[] = "Pragma: "; // browsers keep this blank.\n\n\n        curl_setopt($this->curl, CURLOPT_USERAGENT, \'Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7\');\n        curl_setopt($this->curl, CURLOPT_HTTPHEADER, $header);\n        curl_setopt($this->curl,CURLOPT_COOKIEJAR, $this->cookieJar); \n        curl_setopt($this->curl,CURLOPT_COOKIEFILE, $this->cookieJar);\n        curl_setopt($this->curl,CURLOPT_AUTOREFERER, true);\n        curl_setopt($this->curl,CURLOPT_FOLLOWLOCATION, true);\n        curl_setopt($this->curl,CURLOPT_RETURNTRANSFER, true);  \n    }\n\n\n    function get($url)\n    { \n        $this->curl = curl_init($url);\n        $this->setup();\n\n        return $this->request();\n    }\n\n    function getAll($reg,$str)\n    {\n        preg_match_all($reg,$str,$matches);\n        return $matches[1];\n    }\n\n    function postForm($url, $fields, $referer=\'\')\n    {\n        $this->curl = curl_init($url);\n        $this->setup();\n        curl_setopt($this->curl, CURLOPT_URL, $url);\n        curl_setopt($this->curl, CURLOPT_POST, 1);\n        curl_setopt($this->curl, CURLOPT_REFERER, $referer);\n        curl_setopt($this->curl, CURLOPT_POSTFIELDS, $fields);\n        return $this->request();\n    }\n\n    function getInfo($info)\n    {\n        $info = ($info == \'lasturl\') ? curl_getinfo($this->curl, CURLINFO_EFFECTIVE_URL) : curl_getinfo($this->curl, $info);\n        return $info;\n    }\n\n    function request()\n    {\n        return curl_exec($this->curl);\n    }\n}\n\n?>\n</code>\n</pre>\n'}, {'answer_id': 105929, 'author': 'troelskn', 'author_id': 18180, 'author_profile': 'https://Stackoverflow.com/users/18180', 'pm_score': 2, 'selected': False, 'text': '<p>If you need something that is easy to maintain, rather than fast to execute, it could help to use a scriptable browser, such as <a href="http://simpletest.org/en/browser_documentation.html" rel="nofollow noreferrer">SimpleTest\'s</a>.</p>\n'}, {'answer_id': 1962728, 'author': 'Sarfraz', 'author_id': 139459, 'author_profile': 'https://Stackoverflow.com/users/139459', 'pm_score': 0, 'selected': False, 'text': '<p>Scraper class from my framework:</p>\n\n<pre><code>&lt;?php\n\n/*\n    Example:\n\n    $site = $this-&gt;load-&gt;cls(\'scraper\', \'http://www.anysite.com\');\n    $excss = $site-&gt;getExternalCSS();\n    $incss = $site-&gt;getInternalCSS();\n    $ids = $site-&gt;getIds();\n    $classes = $site-&gt;getClasses();\n    $spans = $site-&gt;getSpans(); \n\n    print \'&lt;pre&gt;\';\n    print_r($excss);\n    print_r($incss);\n    print_r($ids);\n    print_r($classes);\n    print_r($spans);        \n\n*/\n\nclass scraper\n{\n    private $url = \'\';\n\n    public function __construct($url)\n    {\n        $this-&gt;url = file_get_contents("$url");\n    }\n\n    public function getInternalCSS()\n    {\n        $tmp = preg_match_all(\'/(style=")(.*?)(")/is\', $this-&gt;url, $patterns);\n        $result = array();\n        array_push($result, $patterns[2]);\n        array_push($result, count($patterns[2]));\n        return $result;\n    }\n\n    public function getExternalCSS()\n    {\n        $tmp = preg_match_all(\'/(href=")(\\w.*\\.css)"/i\', $this-&gt;url, $patterns);\n        $result = array();\n        array_push($result, $patterns[2]);\n        array_push($result, count($patterns[2]));\n        return $result;\n    }\n\n    public function getIds()\n    {\n        $tmp = preg_match_all(\'/(id="(\\w*)")/is\', $this-&gt;url, $patterns);\n        $result = array();\n        array_push($result, $patterns[2]);\n        array_push($result, count($patterns[2]));\n        return $result;\n    }\n\n    public function getClasses()\n    {\n        $tmp = preg_match_all(\'/(class="(\\w*)")/is\', $this-&gt;url, $patterns);\n        $result = array();\n        array_push($result, $patterns[2]);\n        array_push($result, count($patterns[2]));\n        return $result;\n    }\n\n    public function getSpans(){\n        $tmp = preg_match_all(\'/(&lt;span&gt;)(.*)(&lt;\\/span&gt;)/\', $this-&gt;url, $patterns);\n        $result = array();\n        array_push($result, $patterns[2]);\n        array_push($result, count($patterns[2]));\n        return $result;\n    }\n\n}\n?&gt;\n</code></pre>\n'}, {'answer_id': 3075685, 'author': 'adiian', 'author_id': 268856, 'author_profile': 'https://Stackoverflow.com/users/268856', 'pm_score': 1, 'selected': False, 'text': '<p>here is another one: a simple <a href="http://php-html.net/tutorials/how-to-write-a-simple-scrapper-in-php-without-regex/" rel="nofollow noreferrer">PHP Scraper without Regex</a>.</p>\n'}, {'answer_id': 3784264, 'author': 'Joe Niland', 'author_id': 366965, 'author_profile': 'https://Stackoverflow.com/users/366965', 'pm_score': 4, 'selected': False, 'text': '<p><a href="http://scraperwiki.com/" rel="noreferrer">ScraperWiki</a> is a pretty interesting project.\nHelps you build scrapers online in Python, Ruby or PHP - i was able to get a simple attempt up in a few minutes.</p>\n'}, {'answer_id': 10763495, 'author': 'Salman von Abbas', 'author_id': 362006, 'author_profile': 'https://Stackoverflow.com/users/362006', 'pm_score': 4, 'selected': False, 'text': '<p>I recommend <a href="https://github.com/fabpot/Goutte" rel="noreferrer">Goutte, a simple PHP Web Scraper</a>.</p>\n\n<h3>Example Usage:-</h3>\n\n<p>Create a Goutte Client instance (which extends\n<code>Symfony\\Component\\BrowserKit\\Client</code>):</p>\n\n<pre><code>use Goutte\\Client;\n\n$client = new Client();\n</code></pre>\n\n<p>Make requests with the <code>request()</code> method:</p>\n\n<pre><code>$crawler = $client-&gt;request(\'GET\', \'http://www.symfony-project.org/\');\n</code></pre>\n\n<p>The <code>request</code> method returns a <code>Crawler</code> object\n(<code>Symfony\\Component\\DomCrawler\\Crawler</code>).</p>\n\n<p>Click on links:</p>\n\n<pre><code>$link = $crawler-&gt;selectLink(\'Plugins\')-&gt;link();\n$crawler = $client-&gt;click($link);\n</code></pre>\n\n<p>Submit forms:</p>\n\n<pre><code>$form = $crawler-&gt;selectButton(\'sign in\')-&gt;form();\n$crawler = $client-&gt;submit($form, array(\'signin[username]\' =&gt; \'fabien\', \'signin[password]\' =&gt; \'xxxxxx\'));\n</code></pre>\n\n<p>Extract data:</p>\n\n<pre><code>$nodes = $crawler-&gt;filter(\'.error_list\');\n\nif ($nodes-&gt;count())\n{\n  die(sprintf("Authentification error: %s\\n", $nodes-&gt;text()));\n}\n\nprintf("Nb tasks: %d\\n", $crawler-&gt;filter(\'#nb_tasks\')-&gt;text());\n</code></pre>\n'}, {'answer_id': 28095262, 'author': 'PHP Addict', 'author_id': 4420216, 'author_profile': 'https://Stackoverflow.com/users/4420216', 'pm_score': 2, 'selected': False, 'text': '<p>Scraping can be pretty complex, depending on what you want to do. Have a read of this tutorial series on <a href="http://php-addict.com/write-basic-scraper-php-tutorial-1/" rel="nofollow">The Basics Of Writing A Scraper In PHP</a> and see if you can get to grips with it.</p>\n\n<p>You can use similar methods to automate form sign ups, logins, even fake clicking on Ads! The main limitations with using CURL though are that it doesn\'t support using javascript, so if you are trying to scrape a site that uses AJAX for pagination for example it can become a little tricky...but again there are ways around that!</p>\n'}]
['https://Stackoverflow.com/questions/26947', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/2052/']