The Web as a source for data

New forms of data

• New forms of data enabled by web technologies
• Digital life, digital traces, digitalised life (Lazer & Radford, 2017)
• Much of these data are available as an accident (Arribas-Bel, 2014)

Sources

Arribas-Bel (2014):

• Citizens as sensors
• Businesses moving online
• Open (government) data

“Citizens as sensors”

• Web 2.0 –> (data) consumers are also creators (Goodchild, 2007)
• Much of everyday life now recorded in a database
• (Parts of) these databases can be accessed

“Businesses moving online”

• Business activities digitalised (or entirely digital)
• Records stored in databases, that can also be accessed
• Active/passive collection (eg. IoT)

Governments “opening up”

• Public institutions releasing internal data under open licenses
• Variety of reasons (transparency, economic value, service…)
• Not to take for granted!

Challenges

• Bias
• Licensing
• Access

Bias

• Who’s represented?
• Who’s missing?
• How is the data used?

Licensing

• How accessible data can be used, what for
• “Free beer”, not “free speech”
• Evolving landscape

Access

• Vehicles used to make data available are programmatic (APIs)
• Mostly based on web technologies
• Additional set of skills required

Web APIs

Web APIs

Programmable interfaces to web services

• Machine-to-machine communication over the web
• Expose access to (parts of) databases, or output of computations
• Bridge between client and server

How APIs work

• Authenticate (potentially linked to billing) [Optional]
• Send query as URL
• Receive request ((Geo)JSON, .csv)

Web APIs: an example

Over HTTP (e.g. REST): the query is a URL:

https://services1.arcgis.com/
                ESMARspQHYMw9BZ9/arcgis/rest/services/
                LSOA_DEC_2011_EW_BSC/FeatureServer/0/
                query?
                where=1%3D1&
                outFields=*&
                outSR=4326&
                f=json

[URL]

Web APIs: language “bindings”

• Library in a programming language (e.g. Javascript, Python, R) to interact with an API
• Wrapper in the language around HTTP endpoints
• Expose API in familiar form for developers of that language

Web APIs: some thoughts

• Very useful to access new forms of data
• All different, but some patterns are broadly general
• Programming skills help to automate data access

API examples

API examples

Access to:

• Data
• Maps
• Computations

http://tile.stamen.com/terrain/{z}/{x}/{y}.jpg