My initial thought was to follow a model similar to this new project. But this approach leaves a number of common scraping problems unresolved:

1. No caching. Each time data is required, the code will request the source HTML from the target URL
2. Linear performance. Each time data is required, the code must process the HTML into API objects
3. Relies on well-formed XML. If PHP’s SimpleXML extensions receives tag-soup the solution will fail
4. Complex code to maintain. When the target website changes the structure of their HTML, it means a complete re-write of the majority of the API code

Enter the Reaper

To address these issues, I have been developing Reaper. Currently a PHP implementation that doesn’t require any extensions or external libraries. Reaper attempts to condense the common tasks of scraping into small blocks of efficient code and cache the results transparently for best performance:

1. Reaper requests the URL (via YQL). HTML returned is tidied into well-formed XML and cached
2. Reaper accepts your data definition array which maps data labels to XPath queries, RegEx expressions and/or callback functions to scrape the relevant data
3. Reaper caches the resulting data object and returns it to you

There’s more work to do to improve error-handling and documentation, but so far I’m pretty pleased with the results.

Meanwhile, I’ve stumbled onto Steam Condenser, so I may not need to roll my own Steam Community API after all

I’m keen to hear suggestions and feedback, so let me know what you think as a comment, using the contact form or on Twitter.

Recently I’ve decided it’s time to bite the bullet and formalise some of these hacks, so I bring you my first two public WordPress plugins:

• Custom default avatar

  Plain vanilla WordPress provides a list of default avatars to choose from, but doesn’t allow you to choose an image of your own making. This plugin allows you to specify your own default avatar

• Custom app icons

  This plugin allows you to specify icon(s) to be used when iPhone / iPod Touch users create a shortcut to your site using the ‘Add to Home Screen’ function in Safari

I hope you find these useful, and eventually I will think about submitting for inclusion in the WordPress.org plugin directory. Before I do however, I’d appreciate any and all feedback, for example: any functionality limitations, plugin faux pas, coding style issues, etc…

Let me know what you think!

However, not all URLs were created equal. Some URLs are “nicer” than others. For example, URLs with query string parameters are often considered to belong to the “not so nice” URL category: http://example.com/?p=1234&vH=10&Session_ID=er5DKJn838JK2dfs

In general, what I consider to be “nice” or “not so nice” URLs is a lengthy topic, and I’ll only touch on part of it today. Suffice to say, that for some purposes, I believe using query string parameters is not the worst crime you can commit. In fact, in some cases, I believe they are perfectly acceptable.

Take the following URL for instance: http://example.com/books/?format=html&order=alphabetical&page=2. Although query string parameters mean this URL is a little tricky to read, at least it uses human-readable parameter keys and values. And because slashes / in URLs imply heirarchy, the only good alternative for this type of URL would be a Matrix URL, like this: http://example.com/books/;format=html;order=alphabetical;page=2.

Implementing Matrix URLs within web applications can be difficult, requiring extra server-side redirects or client-side trickery because by default, a HTML form won’t submit data formatted as a Matrix URL.

That’s why I believe query strings aren’t so bad, sometimes they really come in handy.

Repeated parameters

That said, when using checkboxes (or heaven-forbid) multi-select controls to submit data using the GET method, some server-side languages (like PHP) require that you add [] to the end of the name attribute of each control, for example: <input type="checkbox" name="items[]" value="item1" /><input type="checkbox" name="items[]" value="item2" />

For my money, this results in “not so nice” URLs, for example: http://example.com/books/?items[]=item1&items[]=item2

I know it’s a subtle difference, but I much prefer: http://example.com/books/?items=item1&items=item2

The other benefit is that your HTML wouldn’t need to contain the [] either: <input type="checkbox" name="items" value="item1" /><input type="checkbox" name="items" value="item2" />

A problem

The problem is, by default, if [] doesn’t appear in your URLs, only the last ‘items’ parameter will be accessible to PHP in the $_GET array.

A solution

I spent some time thinking about this, and decided the best thing to do would be to parse the URL myself.

/**
 * Returns query string parameters more intelligently from the URL than by using the $_GET array.
 *
 * When multiple parameters are encountered with the same name, they are stacked into an
 * array. This means all URL data can be accessed without using brackets in name attributes
 * For example, typically you would use: <input name="items[]" /> resulting in &items[]=id1&items[]=id2
 * However, using this method you can use: <input name="items" /> resulting in &items=id1&items=id2
 *
 * @author Andrew Ramsden
 * @see: http://irama.org/news/2009/10/17/slightly-nicer-urls/
 * @license GNU GENERAL PUBLIC LICENSE (GPL) <http://www.gnu.org/licenses/gpl.html>
 *
 * @param String $url (optional) A URL to parse for query string variables. If not set, the
 *        current requested URI will be parsed.
 * @return Array An associative array with all query string variables. Multiple parameters
 *         are stacked into a nested array.
 */
function getURLVariables ($url='') {
	$url = !empty($url) ? parse_url($url) : parse_url($_SERVER['REQUEST_URI']);
	$result = array();
	$queryStrParams = explode('&',$url['query']);
	foreach ($queryStrParams as $param) {
		$paramKeyVals = explode('=',$param, 2);
		if (!isset($paramKeyVals[0])) continue;
		$key = $paramKeyVals[0];
		$val = isset($paramKeyVals[1])?$paramKeyVals[1]:'';
		if (substr($key,-6) == '%5B%5D') { // support ugly urls too
			$result[substr($key,0,-6)][] = $val;
		} else if (!isset($result[$key])) { // add new param to the results array
			$result[$key] = $val;
		} else { // this param already exists, stack into an array
				if (is_array($result[$key])) {
					$result[$key][] = $val; // add to existing array
				} else {
					$result[$key] = array($result[$key], $val); // create new array
				}
		}
	}
	return $result;
}

Now instead of using: $items = $_GET['items']; you can use $items = getURLVariables()['items']; and access all the data from your slightly nicer URLs.

Feedback appreciated, let me know what you think.

From the PHP manual:

…Also note that it is your responsibility to die() if necessary…

Obviously I missed the memo where PHP developers were sworn to a secret pact, where among other duties, we are required to perform acts of international espionage and die() if neccessary.

I guess it came between the whitepapers karate chop(): land the death blow and Rigged to explode(): Dangerous things 101.