iSpecies

Offline

2008-06-10T23:29:00.001+01:00

iSpecies was off-line for a few hours today. I moved it from a local folder in my user folder to the /Library/Server folder on the web server, and associated ispecies.org with it's own IP address (although it is still served from the same machine). Glasgow University's DNS seems takes a while to update, so consequently the site appeared to be broken. A quick external check using Network-Tools.com confirmed that ispecies.org had the new IP address, but locally it was still resolving to the holding page of 123-reg, with whom I registered the domain. By fussing with the VirtualHost directive in the Apache httpd.conf file, I managed to get it working again.

NameVirtualHost 130.209.46.63
<VirtualHost 130.209.46.63>   
   DocumentRoot "/Library/WebServer/ispecies"
   ServerName ispecies.org
   ServerSignature email
   DirectoryIndex index.php index.html index.htm index.shtml
   LogLevel warn
   HostNameLookups off
   <Directory "/Library/WebServer/ispecies">
      allow from all
      Options +Indexes
   </Directory>  
</VirtualHost>

The only difference users may notice is that the URLs will now always start with http://ispecies.org.

Wikipedia on iSpecies

2008-03-25T16:04:00.004Z

I've added snippets from Wikipedia to iSpecies results, in part inspired by FreeBase. This makes use of the XML export format . For example, the URL http://en.wikipedia.org / wiki / Special:Export / Luzon_Montane_Forest_Mouse returns XML, with the wiki markup enclosed in the tags <text xml:space="preserve"></text> I use some simple regular expressions to strip some of the markup out, including the taxobox, then I grab the first 100 words of the article to display on the iSpecies page (together with a link to the original article).

Because a species may have multiple names, we need to handle redirection. For example, the URL http://en.wikipedia.org / wiki / Special:Export / Apomys_datae returns

<text xml:space="preserve">#Redirect [[Luzon Montane Forest Mouse]]</text>

which tells us that the content is to be found at http://en.wikipedia.org / wiki / Special:Export / Luzon_Montane_Forest_Mouse.

There's still some polishing to do, but the Wikipedia snippets add something to the iSpecies results.

Maps, and a Google tweak

2007-08-30T15:22:00.000+01:00

Today I stumbled across the Species Distribution Widget from GBIF (written by Tim Robertson and Dave Martin). For Mac OS X 10.4 users, this provides a cool way to quickly get a distribution map for a taxon. Given that Apple dashboard widgets are essentially Javascript and HTML, it occurred to me to reverse engineer the widget to see what it did. To open the widget you just "Ctrl-click" on the widget icon, select Show Package Contents, and the contents open in a Finder window.

The guts of the widget is in the scripts folder. This contains a Javascript file. The widget calls the URL http://data.gbif.org/species/taxonName/ajax/returnType/concept /view/ajaxMapUrls/provider/1/?query=, to which is appended the taxon name you are searching for. Back comes the result in XML. For example, searching for Apus apus returns:

<taxons>
 <taxon>
  <name>Apus apus</name>
   <commonName>Common swift</commonName>
   <key>13836131</key>
   <url>species/13836131/overviewMap.png</url>
 </taxon>
</taxons>

(Shouldn't "taxons" be "taxa"?) The URL of the corresponding map is given in the <url> tag. Append this to "http://data.gbif.org/, and you have the URL for the image of the map. For example, here's the map for Apus apus.

I've added code to do this to iSpecies, so it now features maps from from GBIF. I've also finally tweaked the Google code to stop mangling UTF-8 characters.

5 Ways to Mix, Rip, and Mash Your Data

2007-03-05T15:38:00.000Z

Spotted by Simon Rycroft, Nick Gonzalez has a comparison of maship scripts entitled: 5 Ways to Mix, Rip, and Mash Your Data.

Call them pipes, teqlos, dapps, modules, mashups or whatever else but fact is that recently we have seen a good number of new services that allow developers and users to build mini-apps and mashups that mix and re-mix data. Here we run through 5 applications that allow you to mix, rip and mash your data, looking at the data input, output, REST support, suggested use, and required skill level.

Clearly, this stuff is attracting a lot of attention.

Wikis and the future of iSpecies

2007-03-03T19:07:00.000Z

So, where next for iSpecies? An obvious route seems to be adding a Wiki, something I've discussed on SemAnt. Imagine pre-populating a Wiki with search results from iSpecies, especially if we drilled down using the links in the NCBI search results to extract further content, and made use of the improved mapping between NCBI and TreeBASE names (TBMap).

A few things have stopped me from implementing this. One is the problem that Wiki's are (usually) just unstructured text. However, semantic wikis are starting to emerge (e.g., Semantic MediaWiki, and Rhizome -- I'll be adding links to more at del.icio.us/rdmpage/semantic-wiki). Using a semantic wiki means we can enter structured information and render it as RDF, which would make it potentially a great way to cpature basic facts (triples) about a taxon, but still have human-readable and editable documents.

I've been pondering this, and toying with either writing something myself, or using an off the shelf solution. It's like that I may write something, because I want to link it to a triple store, and I want to pre-populate the wiki as much as possible.

One minor thing that has been holding me back is thinking about URLs to link to the content. For example, I'd like to be able to do the following:

Link to a page by either a unique numerical identifier (e.g., "wiki/0342001", or a name (e.g., "wiki/Physeter catodon"). If the user enters the numerical version, they get directed to the text identifier.

If a name is a synonym, redirect user to that page. For example, "wiki/Physeter macrocephalus" would redirect to "wiki/Physeter catodon").

If the name is a homonym, display a disambiguation page listing the different taxa with that name.

If a user creates a URL that doesn't exist, the wiki would offer to make a new page, after checking that the URL tag is a scientific name (say by using uBio's XML web service).

I've been learning about the joys of Apache's mode-rewrite, which looks like a nice way to deal with some of these issues. For example, this .htaccess file handles both numerical and text identifiers.


# Don't mess with the actual script call
RewriteRule ^get.php*         -             [L]
# URL is numerical id
RewriteRule (^[0-9]*$)	get.php?id=$1       [L]
# URL is tag name
RewriteRule (^[A-Za-z](.*))	get.php?name=$1 [L]

Then, the code in get.php would do display the appropriate page. If the parameter is a numerical id, it's a simple database lookup (numerical identifiers are great because databases handle them easily, and they can be stored without worrying about issues such as capitalisation and punctionation). If it's a name we follow the steps outlined above to handle synonyms, etc.

The point of this is that we get clean URLs, but users can still link using natural URLs like those in WikiPedia and WikiSpecies. Given this, why don't I use WikiSpecies? Well, because it's not a semantic wiki, so I don't gain anything from locking information up in this format.

RSSBus

2007-02-22T20:38:00.000Z

David Shorthouse altered me to RSSBus, which is similar to Yahoo's Pipes, but Lance Robinson (the "Tech Evangelist" at RSSBus) argues that their product is much better. What is RSSBus?

RSSBus is a Really Simple Service Bus that uses the RSS protocol as the main interchange mechanism. RSS is an extensible protocol used to exchange Feeds of Items. Normally these are news items or blog postings, but they don't have to be: RSS Feeds may be augmented through standard RSS extensions to exchange any type of data.

RSSBus is a collection of tools and services that simplify the process of creating RSS Feeds with rich data extensions. Feeds are generated from RSSBus Connectors, reusable code modules that convert data into feeds. They do so by communicating with RSSBus over defined interfaces (please refer to our RSSBus Connectors Reference for details on building custom connectors).

RSSBus provides an infrastructure for generating, maintaining, combining, manipulating, and visualizing Feeds. Items and Feeds are orchestrated by the RSSBus Engine and together help create a loosely integrated application architecture which we like to refer to as RSS Web.

David says he has managed to recreate iSpecies on his desktop with RSSBus, which sounds cool. So far RSSBus is a Windows only tool, although there is code for other platforms listed on the blog. There is also a white paper.
Looks like the conversation on OpenSearch, RSS, and biodiversity informatics has only just got started.

Open Search and The Nearctic Spider Database - almost there

2006-12-02T15:59:00.000Z

As announced on TAXACOM, David Shorthouse has added an Open Search interface to his really nice Nearctic Spider Database. As I've noted previously (see Adding sources to iSpecies and OpenSearch and iSpecies ), OpenSearch seems an obvious candidate for a simple way to add search functionality to biodiversity web sites.

The interface is generated by some software called Zoom Search, and the interface is here. As an example, here is a query for the spider Enoplognatha latimana.

But...

Having an easy way to search a site using a URL API such as Open Search is great, but the feed is RSS 2.0, and as a result has very little information. For example, here's an extract:

<item>
<title>The Nearctic Spider Database: Enoplognatha latimana Hippa & Oksala, 1982 Description</title>
<link>http://canadianarachnology.dyndns.org/data/spiders/7561</link>
<description>THERIDIIDAE: Enoplognatha latimana taxonomic and natural history description in the Nearctic Spider Database.</description>
<zoom:context> ... Descriptions Home Search: Register Log in Enoplognatha latimana Hippa& Oksala, 1982 Temporary ... 2007 Arachnid Calendar FAMILY: THERIDIIDAE Sundevall, 1833 Genus: Enoplognatha Pavesi, 1880 ...</zoom:context>
<zoom:termsMatched>2</zoom:termsMatched>
<zoom:score>1804</zoom:score>
</item>

This information is intended to be displayed in a feed reader, and hence viewed by a human. But, what if I want to put this information in a database, or combine it with other data sources in a mashup, such as iSpecies? Well, I have to scrape information out of free formatted text. In other words, I'm no further forward than if I scraped the original web page.

If we want to make the information accessible to a computer, then we need something else. RDF is the obvious way forward.

The difference that RDF makes

To illustrate the difference, let's search for images of the same spider (Enoplognatha latimana) using my Open Search wrapper for Yahoo's images search (described in OpenSearch and iSpecies). Here is the query. This feed is formatted as RSS 1.0, and I can view it in a feed reader, such as NetNewsWire.

But, because the feed is RSS 1.0 and therefore RDF, the feed contains lots of information on the image in a form that can be easily consumed.

<foaf:Image rdf:about="http://www.spiderling.de/arages/
Fotogalerie/Enoplognatha_latimana_1024.jpg">
<dc:type>image</dc:type>
<dc:title>Enoplognatha_latimana_1024.jpg</dc:title>
<dc:description></dc:description>
<dc:subject>Enoplognatha latimana</dc:subject>
<dc:source>http://www.spiderling.de/arages/
Verbreitungskarten/ENO_LAT0.HTM</dc:source>
<dc:format>image/jpeg</dc:format>
<foaf:thumbnail rdf:resource=
"http://re3.mm-a1.yimg.com/image/206564554"/>
</foaf:Image>

In this example, I use the FOAF and Dublin Core vocabularies. these are widely used, making it easy to integrate this information into a larger database, such as a triple store. To my mind, this is the way forward. We need to move beyond thinking about making data only accessible to people, and making it accessible to computers. Once we do this, then we can start to aggregate and query the huge amounts of data on the web (as exemplified by David's wonderful site on spiders). And once we do that, we may discover all sorts of things that we don't know (see Disconnected databases, and Discovering new things).

Identification service

2006-11-06T17:16:00.000Z

Nick Hobgood emailed me asking whether iSpecies supports requests for identifications. In other words, is this fish Rudarius minutus?

iSpecies doesn't support requests, but it strikes me a useful idea if there was a place where such requests could be directed. The TAXACOM mailing list is one place I've seen requests made, but a mailing list is probably not the best forum. An interesting idea to pursue...

OpenSearch and iSpecies

2006-09-05T11:08:00.000+01:00

I've mentioned OpenSearch in an earlier post, in the context of adding additional sources to iSpecies. But it's slowly dawned on me that what i should be doing is wraping the sources I currently use in OpenSearch as well. Hence, any data source would have a consistent query interface, and a consistent return format. If we ensure the later is RDF, then we get aggregation "for free".

So, I've made a start. First up is Yahoo's image search, which I've wrapped as http://darwin.zoology.gla.ac.uk/cgi-bin/yahoo.cgi. You just append "q=" and the search terms to get a result. Try an example search for images of the ant Atta mexicana. Note that I currently just support the return format, not the query format (that'll come later). The query result is RSS 1.0 because it contains RDF (RSS 2.0 and Atom don't, and hence for my purposes are beside the point). The upshot is that I can now use this search in other projects, and making a better iSpecies becomes simply a case of adding a bunch of OpenSearch sources together.

Generating the RSS proved "fun", but the feed now validates as RDF, although Feed Validator grumbles slightly. It's all a bit of a black art, but I had to nest the RDF payload in <content:item> tags, like this:

<content:item>
<foaf:Image rdf:about="http://www.par...x.draw.JPEG">
  <dc:type>image</dc:type>
  <dc:title>Atta.mex.draw.JPEG</dc:title>
  <dc:description>Leaf-cutter ants (Atta mexicana ) ... </dc:description>
  <dc:subject>Atta mexicana</dc:subject>
  <dc:source>http://www.parasiticplants.siu.edu/Viscaceae</dc:source>
  <dc:format>image/jpeg</dc:format>
  <foaf:thumbnail rdf:resource="http://mud.mm-a5.yimg.com/image/2050519657"/>
</foaf:Image>
</content:item>

More DOIs

2006-09-01T16:08:00.000+01:00

Following on from an earleir post, I've now added DOI extraction for SciELO, which hosts Brazilian publications, and Taylor and Francis. This was motivated by searching iSpecies for the ant Trachymyrmex opulentus, for which only papers hosted by these two publishers appear in the search results.

Again, we are reduced to screen scraping (sigh). Why oh why don't the people who design these web sites get their act together and embed useful information in the HTML, rather than assume that only humans will make use of these pages?

One provider that is clued up is Ingenta. For example, take a look at the HTML for the article "Influence of Topography on the Distribution of Ground-Dwelling Ants in an Amazonian Forest" (doi:10.1076/snfe.38.2.115.15923) on the Ingenta site (Firefox and Camino users can see the source here). Embedded in the <meta> tags is all sorts of metadata, including the DOI:


<meta name="DC.identifier" scheme="URI" 
   content="info:doi/10.1076/snfe.38.2.115.15923"/>

The use of consistently formatted tags makes data extraction much easier. Of course, it's no surprise that Ingenta do this well (check out their blog).

Extracting DOIs

2006-08-29T22:31:00.000+01:00

DOIs are pretty cool, so I spent a little time this evening working out how to extract DOIs from Google Scholar results for journals hosted by Springer, JStor, and J-Stage I've also added code to extract Serial Item and Contribution Identifiers (SICIs) from JSTor URLs. SICI is NISO standard Z39.56.

The point of this exercise is to try and get DOIs for as many articles as possible, because DOIs are the GUID of choice for publications, and we can extract metadata for a DOI, either directly using CrossRef's OpenURL resolver, or via Connotea. This will make life easier for the next step, namely aggregating literature into a triple store.

Add to Connotea

2006-08-09T17:11:00.000+01:00

Finally go around to adding a "Add to Connotea" button to the Google Scholar results, based on code from Postgenomic. The code is a simple bit of Javascript:


<a style="cursor:pointer;" onclick="javascript:
u='http://dx.doi.org/10.1111/j.1744-7429.2005.37_04_01.x';
a=false;
x=window;
e=x.encodeURIComponent;
d=document;
w=open('http://www.connotea.org/addpopup?continue=confirm
&uri='+e(u),'add','width=660,height=300,scrollbars,resizable');
void(x.setTimeout('w.focus()',200));">
<img src="images/connotea.png" border="0" 
alt="add bookmark to connotea" align="absmiddle"></a>

where http://dx.doi.org/10.1111/j.1744-7429.2005.37_04_01.x is the URI of the article being added.

Now a click brings up Connotea and you can add a paper you've found using iSpecies. At present this only works for papers where I've extracted a DOI.

Adding sources to iSpecies

2006-07-08T01:47:00.000+01:00

One issue which comes up every so often is how to add data sources to iSpecies. At present iSpecies queries NCBI, Yahoo images, and Google Scholar, each source requiring different code to make the query and handle the response. If adding new sources requires writing code specific to that source then iSpecies would rapidly become a nightmare (leaving aside the issue that until iSpecies is multithreaded, adding additional sources slows everything down -- see my earlier post about the need for speed).

One solution is to develop a standard search interface and ask data source to adopt that. The obvious candidate is OpenSearch, which I've already touched on over at iPhylo. OpenSearch is appealing because it is no more difficult than serving RSS feeds, and because it is based on RSS it can be integrated into a range of tools, such as Amazon's A9, and Internet Explorer 7.

At a minimum, it would be useful if sources supported OpenSearch. It would also be useful if they supported RSS to serve individual records. This is handy because NCBI links to numerous sources via LinkOut, and hence we could avoid the overhead of doing a search if we can retrieve the record directly (i.e., if NCBI has a link then I already now the information exists).

In say "RSS", I should stress that I really mean RSS 1.0 (i.e., RDF). RSS 2.0 and Atom are a lot less useful in the long run, because RSS 1.0 can be integrated into a triple store, which opens up a world of cool things (i.e., aggregating data and performing queries on that data).

Why Google is good for science...maybe

2006-05-22T11:34:00.000+01:00

I just noticed this piece written by Jeff Perkel in January who, after "poking around" the iSpecies blog, wrote Why Google is good for science. Well, yes and no. On the one hand it's fabulous, but on the other hand they can play rough. For example, iSpecies used Google Scholar to find scientific papers for a species name. The traffic was pretty minimal in the scheme of things, but Google have now blocked iSpecies (and as a consequence my whole University - gulp!) from accessing Google Scholar.

Before anybody says, "but you got what you deserved because you broke Google's Terms of Service", I think in this case they are simply being lazy. If Google truly cared about making Google Scholar useful, they'd create an API. Because they haven't I had to resort to screen scraping their unbelievably awful HTML (and I'm not the only one). The cost of setting up an API along the lines of the one available for the main Google search engine would be trivial.

After venting my spleen, the reason -- as I should of guessed -- is "intellectual property". Google Scholar's agreements with the publishers that they index prevents Google from making it available other then through the web site. Thanks to Rebecca Shapley for clarifying this. Once again, scientists are being ill served by our publishers. Perhaps somebody needs to set up an Open Source/Open Access equivalent of Google Scholar.

This is what I originally wrote, which perhaps is another reason publishers don't want Google Scholar having an API:

There would also be a potential market. In the UK we rate our research based on a number of factors including journal impact factor, as part of the gargantuan Research Assessment Exercise. Impact factors are supplied by ISI, and Google Scholar results compare well with that source. Just think of the possibilities of a service that used Google Scholar to rate scientists' output. It could even be part of a service like LinkedIn, whioch I stumbled on via Pierre's blog on geotagging RSS feeds (which is a whole separate issue).

iSpecies down

2006-04-24T12:09:00.000+01:00

As reported on iPhylo, the machine running iSpecies was hacked last week. It's taking a while to rebuild things, but iSpecies is now running on another machine until the hacked machine can be rebuilt. My apologies for any inconvenience. Apart from issues of backing up, things take time to restore because the original machine ran an old version (4.2) of PHP, and the new machine uses PHP 5.

Building the encyclopedia of life

2006-03-19T19:24:00.001Z

iSpecies is very limited in the sources it uses, and also in what it extracts from its sources. The sources it does query contain a wealth of information. As an example, GenBank sequence AF131710 from Ligophorus mugilinus has the following information about this animal:


FEATURES             Location/Qualifiers
     source          1..374
                     /organism="Ligophorus mugilinus"
                     /mol_type="genomic DNA"
                     /specific_host="Mugil cephalus"
                     /db_xref="taxon:92200"
                     /country="France"

Note the tags "/specific_host" and "/country". By parsing this record we learn that this organism is found in France, and is hosted by Mugil cephalus.

In the same way, the Google Scholar results could be more effectively used. In many cases we could follow the links to get abstracts of articles, then use literature data mining techniques (e.g., Hirschman et al.) to extract information on the organism's ecology, etc.

Extracting this sort of information would be an one way to automate the construction of an encyclopedia of life.

Towards a faster iSpecies: building libxml and libxslt on Mac OS X

2006-03-19T12:07:00.000Z

iSpecies is written in PHP, and calls a Perl CGI script (to query Google Scholar). This works, but is a bit slow. It also puts limits on what we can do. For example, it would be cool to make the search multithreaded so that the different sources are queried at the same time. This becomes a major issue if we want to "drill down." For example, if a taxon exists in NCBI, it would be useful to visit all the LinkOut resources and collect whatever information they make available. Likewise, Google Scholar results contain links to publishers that could be explored further (such as extracting bibliographic information from RIS files, or RSS feeds such as those available for Ingenta-hosted journals). All of this would delay displaying search results to the user, especially if we have to visit one link after another.

Multithreading would help, but PHP doesn't do this, hence I'm toying with moving to C++ and building a "proper" application (I don't do Java). This means I need to get XML, XPath, and XSLT libraries for C/C++, and this has been, ahem, interesting. Was going to use Sablotron (which I use in my PHP 4 and Perl work), but its documentation is just awful (where are some nice examples?). Will probably use libxml and libxslt. These come with Mac OS X 10.3.9 (I do my development on a G4 iBook, before moving stuff to a Linux box), but Apple hasn't compiled libxml with XPath support (sigh). I built libxml 2.2.63 OK, but libxslt 1.1.15 needed a little hand holding because of the presence of Apple's libxml. The following does the trick:


./configure --with-libxml-prefix=/usr/local

This tells configure to use the version of libxml I installed in /usr/local. Now, once I get my head around libcurl I'll try and build something and see if we can speed up iSpecies.

Silobreaker

2006-02-27T13:32:00.000Z

Silobreaker looks to be a very cool way of exploring information. Facetted browsing is an old idea, but this looks like it might actually make it fun.

(Via Really Simple Sidi (RSS).)

Programmable Web

2006-02-07T13:49:00.000Z

iSpecies makes it onto Programmable Web. This site has all sorts of useful information on Web 2.0 and mashups, see also Mashup Feed. Spotted by Simon Rycroft.

Automatic extraction of references from a paper

2006-01-29T23:06:00.000Z

One goal for iSpecies would be integrating taxonomic literature into the output. This has been motivated by Donat Agosti's efforts to make the taxonomic literature for ants available (see his letter to Nature about copyright and biopiracy doi:10.1038/439392a). For example, we can take a paper marked up in an XML schema such as the TaxonX Treatment Markup, extract the treatments of a name, and insert these into a triple store that iSpecies can query. For a crude example search iSpecies for the "Google ant" Proceratium google.

Now, marking up documents by hand (which is what Donat does) is tedious in the extreme. How can we automate this? In particular, I'd like to automate extracting taxonomic names, and references to other papers. The first can be facilitated by taxonomic name servers, particularly uBio's FindIT SOAP service. Extracting references seems more of a challenge, but tonight I stumbled across ParaCite, which looks like it might do the trick. There is Perl code available from CPAN (although when I tried this on Mac OS X 10.3.9 using cpan it failed to build) and from the downloads section of ParaCite. I grabbed Biblio-Citation-Parser-1.10, installed the dependencies via cpan, then built Biblio::Citation::Parser, and so far it looks promising. If references can be readily extracted from taxonomic markup, then this tool could be used to extract the bibliographic information and hence we could look up the references, both in taxon-specific databases such as AntBase, but also in Google Scholar.

Google Maps Mania: North American Bird Watching Google Maps Mashup

2006-01-25T02:20:00.000Z

Google Maps Mania: North American Bird Watching Google Maps Mashup notes the very slick combination of Google Maps and Flash to display ranges of North American birds at GeoBirds.com. The mashup uses data from the USGS Breeding Bird Survey and the Audobon Society's Christmas Bird Count.

Google, Yahoo, and the death of taxonomy?

2006-01-23T11:31:00.000Z

I posted this on my iPhylo blog, but since it is more relevant to iSpecies, and indeed the talk is the reason I built iSpecies, maybe it belongs here (see, I'm so self-absorbed I've started to blog my blogs - sad).

Wednesday December 7th I gave a talk at the Systematics Association's AGM in London, with the slightly tongue in cheek title Google, Yahoo, and the end of taxonomy?. It summarises some of the ideas that lead me to create iSpecies.org.

For fun I've made a Quicktime movie of the presentation. Sadly there is no sound. Be warned that if you are offended by even mild nudity, this talk is not for you.

The presentation style was inspired by Dick Hardt's wonderful presentation on Identity 2.0.

Identifiers for publications

2006-01-20T17:48:00.000Z

Despite my enthusiasm for LSIDs, here are some thoughts on indentifiers for publications. Say you want to set up a bibliographic database. How do you generate stable identifiers for the contents?

There's an interesting -- if dated -- review by the National Library of Australia.

The Handle System generates Globally Unique Identifiers (GUIDs), such as hdl:2246/3615 (which can be resolved in Firefox if you have the HDL/DOI extension). Handles can also be resolved with URLs, e.g. http://digitallibrary.amnh.org/dspace/handle/2246/3615 and http://hdl.handle.net/2246/3615. DSpace uses handles.

DOIs deserve serious consideration, despite costs, especially if the goal is to make literature more widely available. With DOIs, metadata will go into CrossRef, and publishers will be able to use that to add URLs to their electronic publications. That means people reading papers online will have immediate access to the papers in your database. Apart from cost, copyright is an issue (is the material you are serving copytighted by sombody else?), and recent papers will already have DOIs. Having more than one is not ideal.

If Handles or DOIs aren't what you want to use, then some sort of persistent URL is an option. Their content can be dynamically generated even if they look like static URLs. For background see Using ForceType For Nicer Page URLs - Implementing ForceType sensibly and Making "clean" URLs with Apache and PHP. To do this in Apache you need a .htaccess file in the web folder, e.g.:


# AcceptPathInfo On is for Apache 2.x, don't use for Apache 1.x
<Files uri>
#   AcceptPathInfo On
    ForceType application/x-httpd-php
</Files>

You need to ensure that .htaccess can override FileInfo, e.g. have this in httpd.conf:


<Directory "/Users/rpage/Sites/iphylo">
    Options Indexes MultiViews
    AllowOverride FileInfo
    Order allow,deny
    Allow from all
</Directory>

This would mean that http://localhost/~rpage/iphylo/uri/234 would execute the file uri (which does not have a PHP extension). The file would look something like this:


<?php

 // Parse URL to extract URI
 $uri = $_SERVER["SCRIPT_URL"];
 $uri = str_replace ($_SERVER["SCRIPT_NAME"] . '/', '', $uri);
 
 // Check for any prefixes, such as "rdf" or "rss" which will flag the
 // format to return
 // Check that it is indeed a URI
 // Lookup in our database
 // Display result
 ?>

Lastly, ARK is another option, which is essentially a URL but it copes with the potential loss of a server. It comes from the California Digital Library. I'm not sure how widely this has been adopted. My sense is it hasn't been, although the Northwest Digital Archives is looking at it.

If cost and hassle are a consideration, I'd go for clean URLs. If you wanted a proper bibliographic archive system I'd consider setting up a DSpace installation. One argument I found interesting in the Australian review is that Handles and DOIs resolve to a URL that may be very different to the identifier, and if people copy the URL in the location bar they won't have copied the GUID, which somewhat defeats the point. In other words, if they are going to store the identifiers, say in a database, they need to get the identifier, not the URL.

Creative Commons Welcome Pack

2006-01-20T14:15:00.000Z

Creative Commons Welcome Pack
Originally uploaded by Carlfish.

I want one!

Using RSS feeds to notify clients when data changes

2006-01-20T14:09:00.000Z

iChating with David Remsen earlier this morning, and he suggested using RSS feeds as a way of data providers "notifying" clients if their data has changed (e.g., if they've added some new names). Nice idea, and frees the client (such as a triple store) from having to download the entire data set every time, or to compute the difference between the data held locally and the data held by the remote source. Turns out that HTTP conditional GET can be used to tell if something has changed.

So, the idea is that a data source timestamps its data, and when data is modified it adds the modified records to its RSS feed. The data consumer peridically checks the RSS feed, and if it has changed it grabs the feed and stores the new data (which, in the case of a triple store can be easily parsed into a suitable form).

Says it all...

2006-01-17T01:11:00.000Z

Why I avoid discussions of standards like the plague...

EXIF tags

2006-01-16T09:55:00.000Z

Some images come with embedded metadata, such as EXIF tags or XMP. Images from AntWeb are a good example. These tags can be viewed by various programs, such as Adobe Photoshop, or utilities such as EXIF Viewer, seen here.

So, an obvious step would be (assuming we start using a triple store as a backend for iSpecies, and/or provide the results of a query in RDF) would be to extract metadata from EXIF tags. For example, the image http://www.antweb.org/images/casent0100367/casent0100367_p_1_low.jpg of Proceratium google in AntWeb has the following metadata:

File name: casent0100367_p_1_low.jpg
File size: 17811 bytes (0x0, infbpp, 0x)
EXIF Summary:

Camera-Specific Properties:

Camera Software: EXIFutils V2.5.7
Photographer: April Nobile

Image-Specific Properties:

Image Created: 2005:09:27 09:54:34
Comment: Attribution-NonCommercial-ShareAlike Creative Commons License

Other Properties:

Exif IFD Pointer: 196
Exif Version: 2.20

Hence, we could extract the relevant bits (author, date, copyright) and store those. This could be done in bulk using a tool such as ExifTool.

The example of AntWeb does show one weakness of free-text metadata. The image is licensed under "Attribution-NonCommercial-ShareAlike Creative Commons License". ~~I'm assuming this is Attribution-NonCommercial-ShareAlike 2.0, but without a URL it is a faff to work this out.~~ Ah, looking at the AntBase pages for individual specimens, it's actually 1.0. Yes, it's pretty obvious, but it still requires string matching. These things need to be computer readable as well, and versioned (for example, which version of this license was intended?).

For the photographer (such as AntWeb's April Nobile - seen here), it might be useful to create a FOAF file to link to, so that we have metadata about the creator of the images.

From the Blogsphere

2006-01-09T14:27:00.000Z

Some interesting comments on iSpecies and related matters:

Hip Hop Offers Lessons on Life Science Data Integration
(Great title)

Ontogeny: iSpecies.org and Census of Marine Life Update

I searched on Solenopsis invicta and thought the return information was great.

open data and open APIs enable scientific mashups

The biodiversity community is one group working to develop such services. To demonstrate the principle, Roderic Page of the University of Glasgow, UK, built what he describes as a "toy" — a mashup called Ispecies.org (http://darwin.zoology.gla.ac.uk/~rpage/ispecies). If you type in a species name it builds a web page for it showing sequence data from GenBank, literature from Google Scholar and photos from a Yahoo image search. If you could pool data from every museum or lab in the world, "you could do amazing things", says Page.

Loomware

This is too cool - some interesting examples of "Scientific mashups" from Richard Akerman. I followed the iSpecies link and was blown away by the data that was returned. I studied a damselfly called Argia vivida for my graduate degree and way back then finding data on the bug was not always easy. Searching the species in iSpecies bring up a TaxID number to the NCBI Taxonomy Browser, a list of papers from Google Scholar and images from Yahoo Images. This is an excellent example of what we have all been waiting for with the promise of the Web and web services in particular. The concept of a page for every species known is a dream come true for science, so this one is worth watching.

AntWeb-Google Earth Map

2006-01-06T16:24:00.000Z

AntWeb- Google Earth Map
Originally uploaded by GISuser.com.

This is a nice example of the kind of thing that can be done when georeferenced specimen data are readily available. Need to think about doing this for iSpecies.

For more on AntWeb and Google Earth vist here .

Nature on mashups

2006-01-04T22:41:00.000Z

Nature has an article by Declan Butler on "mashups", which mentions iSpecies, and also Donat Agosti's work on AntWeb and Antbase.

The Modern Palimpsest

2005-12-24T18:41:00.000Z

Leigh Dodds included iSpecies in his recent post on the scientific paper as modern palimpest. He writes:

iSpecies is a nice example of a science "mashup" that illustrates an alternative search interface for finding related content. I used the false results that can appear when performing simple keyword searches to reinforce the need for standard identifiers. (The need for a common, scoped identifier for authors, is a particular hobby horse of mine).

Among the false results is the image of oil tanker (among other things) that Yahoo provides when searching for "Apus apus."

Character coding issues with Google Scholar

2005-12-23T17:06:00.000Z

Tanja points out that results for articles with German titles can look awful (e.g., try searching on Erica inflata). This is a problem with Google Scholar, which corrupts the characters. To verify this, do the search directly in Google Scholar. A workaround, if one had time, would be to screen scrape some of the source sites. For example, Springer's web site could be scraped to get the correct title, and a DOI. One more thing for the to do list...

NatureServe

2005-12-23T10:28:00.000Z

NatureServe, a "non-profit conservation organization that provides the scientific information and tools needed to help guide effective conservation action" have announced an XML schema for their proposed web service. NatureServe's focus (I think) is on rare and endangered species in North America, but some of their data and/or schema may be useful for iSpecies.

Science NetWatch

2005-12-16T20:16:00.000Z

Science magazine's NetWatch column for 16 December 2005 mentions iSpecies.

The Simplicity of the Semantic Web

2005-12-01T09:05:00.000Z

The Simplicity of the Semantic Web
Originally uploaded by kinetisonic.

This is simply to remind me of where iSpecies needs to be heading...

Nodalpoint notices

2005-11-06T04:52:00.000Z

iSpecies was blogged by "Greg" at Nodalpoint, which carried some earlier discussion about LSIDs.

Curiously, LSIDs pop up again:

What is also cool is that each species has RDF formated metadata associated with it via an LSID, see here for an example. It would be nice if each species had its own permanent URL, which would be arguably more useful than an LSID, but I won't go there :)

Actually, iSpecies doesn't link to LSIDs for taxon names (that's done through one of my other toys, the Taxonomic Search Engine). Perhaps it's time to link these two toys together?

Being too careful

2005-11-05T11:51:00.000Z

Managed to figure out why the Google Scholar results would sometimes appear and sometimes not. The server hosting iSpecies uses PortSentry to detect and block port scans. PortSentry decided that the Glasgow University proxy was evil. Our proxy has three IP addresses, two of which were blocked by PortSentry. If the DNS resolved the proxy address to a blocked IP, the Google Scholar Perl script would fail (as it connects to the outside world via the proxy). If the DNS happened to resolve it to the unblocked IP, it would work. Found this out by changing

use LWP;

use LWP::Debug '+';

and printing out the response status_line.

iSpecies launched

2005-11-05T11:34:00.000Z

iSpecies is a very simple test of E O Wilson's idea of a web page for each species. The data displayed are generated "on the fly" by querying other data sources, such as NCBI, Yahoo Images, and Google Scholar. The site was announced on TAXACOM and the Taxonomic Databases Working Group - Structure of Descriptive Data lists on 2 November 2005. It was blogged by Leigh Dodds and Danny Ayers.