Publishing 5-star Open Data

Original data (HTML): schedule, presenters

Cost and benefits of ★ Web data

schedule.pdf

To be honest, this PDF was exported from Excel, which is more than one star. But organisations really often “publish” data in PDF.

Cost and benefits of ★★ Web data

schedule.xls

Even though the old binary *.xls format is proprietary, it is not impossible to read this file outside of Excel:

perl -MSpreadsheet::ParseExcel -le '
  print Spreadsheet::ParseExcel->new()
    ->parse("2star_Excel/schedule.xls")
    ->worksheet("Schedule")
    ->get_cell(1,0)
    ->value();'

Output:

25 Aug 2014 09:00

It is harder, but still possible, to have questions answered such as “when is the first coffee break”.

Think of an algorithm that does the following:

1. In the column titled “Event”, identify all cells whose value is “Coffee break”.
2. On each row of such a cell, get the entry of the cell in the “Time” column.
3. Sort these cells and return the smallest value.

However, free software libraries do not support all features of this file format. Here is what happens when we ask a popular free tool to determine the type of this file:

file 2star_Excel/schedule.xls

Output:

2star_Excel/schedule.xls: Composite Document File V2 Document, corrupt: Can't read SSAT

Cost and benefits of ★★★ Web data

schedule.ods

We can process this file with standard tools:

unzip -l 3star_OpenDocument/schedule.ods

Archive:  3star_OpenDocument/schedule.ods
  Length      Date    Time    Name
---------  ---------- -----   ----
       46  08-21-2014 08:13   mimetype
    52832  08-21-2014 08:13   Thumbnails/thumbnail.png
    27279  08-21-2014 08:13   styles.xml
    15227  08-21-2014 08:13   content.xml
      852  08-21-2014 08:13   meta.xml
     8774  08-21-2014 08:13   settings.xml
      899  08-21-2014 08:13   manifest.rdf
        0  08-21-2014 08:13   Configurations2/accelerator/current.xml
        0  08-21-2014 08:13   Configurations2/progressbar/
        0  08-21-2014 08:13   Configurations2/statusbar/
        0  08-21-2014 08:13   Configurations2/images/Bitmaps/
        0  08-21-2014 08:13   Configurations2/floater/
        0  08-21-2014 08:13   Configurations2/toolbar/
        0  08-21-2014 08:13   Configurations2/popupmenu/
        0  08-21-2014 08:13   Configurations2/toolpanel/
        0  08-21-2014 08:13   Configurations2/menubar/
     1093  08-21-2014 08:13   META-INF/manifest.xml
---------                     -------
   107002                     17 files

content.xml contains the actual tabular data, so we can process it using XPath/XQuery/XSLT tools such as Zorba:

zorba --serialize-text -q '
  declare namespace office="urn:oasis:names:tc:opendocument:xmlns:office:1.0";
  declare namespace table="urn:oasis:names:tc:opendocument:xmlns:table:1.0";
  doc("3star_OpenDocument/content.xml")//office:spreadsheet
    /table:table[@table:name="Schedule"]
    /table:table-row[2]/table:table-cell[1]'

25 Aug 2014 09:00

LibreOffice even stored this timestamp in a machine-friendly way. We'll realise the advantages of this later.

zorba --serialize-text -q '
  declare namespace office="urn:oasis:names:tc:opendocument:xmlns:office:1.0";
  declare namespace table="urn:oasis:names:tc:opendocument:xmlns:table:1.0";
  string(doc("3star_OpenDocument/content.xml")//office:spreadsheet
    /table:table[@table:name="Schedule"]
    /table:table-row[2]/table:table-cell[1]/@office:date-value)'

2014-08-25T09:00:00

We need one CSV file per sheet:

• schedule.csv
• presenters.csv

Cost and benefits of ★★★★ Web data

From here onwards, the original 5-star open data examples use RDF. We will continue with CSV for a while, taking it to its limits, to point out that open data on the Web is not only RDF. We will introduce RDF in a later section.

The following examples roughly conform to Linked CSV, which was one of the original proposals for an RDF-conforming specification of CSV. The CSV on the Web Working Group is now taking a different approach. Their Working Draft on Generating RDF from Tabular Data on the Web suggests leaving the CSV untouched but providing complementary, external metadata annotations, e.g., in the form of JSON. This tutorial sticks with the simpler Linked CSV approach, which is self-contained in CSV.

• schedule.csv
• presenters.csv

Let's continue to make our CSV even more self-describing, by introducing a schema (also called vocabulary on the Web of Data, or ontology, especially when it involves more complex formal logic).

• schedule.csv
• presenters.csv

Cost and benefits of ★★★★★ Web data

More widely than CSV, the RDF data model is used for linked data.

Whenever a URI conforms to linked data, you can expect RDF there (usually in the ugly but widely supported RDF/XML encoding).

Let's therefore redo our example in RDF, and discuss some differences from CSV.

• data.ttl (Turtle, human-friendly)
• presenters.rdf, schedule.rdf, vocab.rdf (RDF/XML, widely understood by machines)

(For purely pragmatic reasons, the Turtle, which is what I edit, is all-in-one, whereas the RDF/XML is in split files for easier deployment.)

<#day1intro>
        dct:date "2014-08-25T09:00:00+02:00"^^xsd:date ;
        dct:title "Introduction" .

CSV is based on records (one per row, with a fixed number of columns).

RDF is based on triples (subject–predicate–object statements).

Usually more than one triple belongs to a subject (resource), which is why it's convenient to group them.

Usually every resource has an identifier. (In the CSV, our events didn't have any.)

You can precisely indicate the datatype of an object, but you also have to do it always, except when the datatype is string.

<#day1keynote>
        a wv:Keynote ;
        dct:date "2014-08-25T09:15:00+02:00"^^xsd:date ;
        dct:title "Keynote" ;
        marcrel:pre <http://purl.org/net/wiss2014/presenters/#stefan> .

It's no problem for different resources to have different numbers of properties.

Compare sparsely populated CSV:

#,Time,Event,Type,Presenter,Location
type,dct:date,dct:title,rdf:type,http://id.loc.gov/vocabulary/relators/pre,schema:location
,2014-08-25T09:00:00+02:00,Introduction,,,

On the other hand, the CSV data model has a built-in order, which RDF does not have. Order can be expressed in RDF, but doing so leads to a high complexity. In the specification on Generating RDF from Tabular Data on the Web, compare the “minimal” RDF representation of a CSV table to the “standard” representation that preserves information about the tabular structure.

For one subject and predicate, there can be multiple objects. In the CSV we had to cheat:

,2014-08-26T18:00:00+02:00,Hackathon dinner,http://purl.org/net/wiss2014/vocab/#Dinner;http://purl.org/net/wiss2014/vocab/#Hackathon,,Maison des Élèves
,http://purl.org/net/wiss2014/presenters/#stefan,Stefan Decker,INSIGHT;National University of Ireland,Galway,Ireland

In RDF, that's no problem:

<#day2hackathondinner>
        rdf:type wv:Dinner, wv:Hackathon ;
        dct:date "2014-08-26T18:00:00+02:00"^^xsd:date ;
        dct:title "Hackathon dinner" ;
        schema:location "Maison des Élèves" .

<http://purl.org/net/wiss2014/presenters/#stefan>
        foaf:name "Stefan Decker" ;
        schema:affiliation "INSIGHT", "National University of Ireland" ;
        wv:town "Galway" ;
        wv:country "Ireland" .

Vocabulary definitions are no problem in RDF either, as RDF Schema itself has an RDF-based syntax:

wv:Hackathon
        rdfs:label "hackathon" ;
        rdfs:comment "an event of intensive collaboration on a software project" ;
        rdfs:seeAlso <http://dbpedia.org/resource/Hackathon> .

Here, we introduced a custom prefix to abbreviate the URI of our vocabulary. Here's how prefixes are declared:

@prefix dct: <http://purl.org/dc/terms/> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix marcrel: <http://id.loc.gov/vocabulary/relators/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix schema: <http://schema.org/> .
@prefix wv: <http://purl.org/net/wiss2014/vocab/#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

This is just syntactic sugar, not part of the RDF data model.

Note that the rdfs:seeAlso link points to DBpedia. DBpedia is a linked dataset extracted from Wikipedia.

Additional stars have been suggested for publishing data …

• … that uses standard schemas – we've done this already.
• … whose quality has been checked – our group does research on this.

Also recall that our original use case started from an HTML homepage. With the following standards it's possible to embed linked data into HTML:

• Microformats (very basic)
• Microdata (more powerful; emphasizes syntactic conciseness)
• RDFa (widest support of the RDF data model) – try it with http://rdfa.info/play/!

The idea for this tutorial was inspired by Antoine Zimmermann. The motivation was to prepare something for the 2014 Web Intelligence Summer School “Web of Data” that's not too heavily biased towards RDF.

This summer school was funded by

This document was created with Emacs Org mode.

The icon for highlighting external links is reused from MediaWiki, developed by the Wikimedia Foundation. Its source code is licensed under GPL version 2 and whose homepage content is licensed under the Creative Commons Attribute-ShareAlike 3.0 Unported License.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Here is how you can cite this using BibTeX (BibLaTeX recommended):

@misc{Lange:5StarData2015,
  title = {Publishing 5-star Open Data},
  author = {Christoph Lange},
  note = {Tutorial at the Web Intelligence Summer School ``Answering Questions with the Web''},
  year = {2015},
  date = {2015-09-01},
  venue = {Saint-{\'E}tienne, France},
  url = {http://clange.github.io/5stardata-tutorial/},
}