csv,conf,v1

The W3C CSV on the Web Working Group has been looking at how to more tightly specify CSV and the provision of metadata that makes it possible to validate CSV and convert it to other formats. This talk will describe the work of the Working Group and its current status.

In PeerLibrary we are recognizing a need for an open dataset of all academic publications. Issues preventing building such dataset are multiple: multiple existing sources, many proprietary, conflicting entries, and legal issues with some parts of data (eg. full text). In addition, collaborative curation of this data is not yet possible. I would like to present what we are doing on this topic, but mostly learn from and about others.

Evolutionary biologists aim to make sense of population behaviour in species across the tree of life. However, the collection of animal and plant population data is laborious and costly so analyses that try to generalise across many species are not feasible unless data are shared among researchers, or obtained from the literature. I will report on the 20+ year journey of construction of two databases that collate demographic data from published literature on more than 2000 species with an aim of making it openly available to all. I will briefly outline why these data are important, describe the process of data production, and contemplate the lessons learned along the way.

Lean ETL, unintentional ddos attacks, impossible parses, backdoor APIs, practices in humility and parsing tactics gleamed from three years as a lead parser at Enigma.io, a company that specializes in normalizing siloed public data.

Numerous attempts have been made over the years to standardise data interchange formats. XBRL, ASN.1, X12, and other terrifying acronyms have caused thousands of developers years of pain and suffering. By contrast, CSV is simple, clear, and easy to use. But it can sometimes be useful to know a little more about the data you have at hand, ideally in a format that is machine-readable and standardised.

I work at Open Knowledge, where we have authored a number of simple standards to improve the portability and reusability of CSV-based data formats. I'll be talking about what data packages are, the Tabular Data Package specification, and some of the tools we've built to help with packaging and using data in these formats.

Waste management is neither a sexy nor a scalable business. Most of the time these two facts are good enough to repel all high-tech startups, leaving a vital public service far away from what can be achieved with modern technology.

Recycling centers are overcrowded. People get angry and don't sort their waste correctly increasing the amount of what is buried or burned. Here comes a bunch of hackers, data-scientists and makers dedicated in producing the best open solution ever. From Open Data, they build a predictive model indicating the best time to got to the recycling center. The information is piped to a mobile app that crowdsources where the waste are produced, what recycling center can take care of it and all the alternatives of reuse. And when more precise data is needed, they'll hack a solution to produce it.

Computers used to be the size of entire rooms. While the hardware has gotten smaller, the space the network occupies has gotten bigger: we're surrounded by hidden infrastructure, from cell towers to fiber lines to networked cameras. For the last few months I've been working on a field guide to seeing that infrastructure, using a mix of sometimes unlikely data sources, including spraypainted street markings for excavation work and DOT permits. This is a talk about finding data in unexpected places, why that data matters, and the value of fieldwork in data-making.

Social scientists have access to many electronically available panel series datasets. However, downloading, cleaning, and merging them together is time-consuming and error-prone: for example, using Reinhart and Rogoff's data on the fiscal costs of the financial crisis involves downloading, cleaning, and merging four Excel files with over 70 individual sheets, one for each country’s data. Furthermore, because such datasets are not bundled in a format that is easy to manipulate, many of them are not updated on a regular basis.

In this talk, we introduce the psData package for the R statistical software. This package is being developed under the rOpenGov framework to solve two problems:

1. Time wasted by social scientists downloading, cleaning, and transforming commonly used data sets for their own research
2. Errors introduced by data import and transformation scripts that are written individually and never shared across researchers

The psData package aims to address these problems by distributing easy to use R functions for downloading, cleaning, and merging datasets used by social scientists. The package focuses on panel series data, which are frequently found in political science and macroeconomics. It is hosted on GitHub and can be easily added to and modified by the community, which will allow to fix and patch distributed datasets to all users simultaneously, improving overall data quality.

The team behind the rOpenGov/psData project currently includes contributors from universities in five countries, and many will be present in Berlin at the time of the conference.

GitHub repository: https://github.com/rOpenGov/psData

I see two main issues in the common means of searching across data tables. One issue is that the search is localized to datasets that are published or otherwise managed by a particular entity, and another is that the search mechanism doesn’t use information relating to the tabular structure of these data tables.

We can do better! Comma search indexes as many data tables as you want, regardless of where they’re from, and it uses the tabular structure of these data tables to search for them. You provide a spreadsheet as input and receive a list of spreadsheets in return. In this talk, you’ll learn how it works and how to use it.

Wikipedia can be a powerful open data resource. The official Wikipedia API provides access to page content and metadata but there are other projects that go much further, offering access to structured and Linked Data knowledge bases that enable advanced querying of data extracted from Wikipedia and beyond. Some projects include DBPedia, Freebase, Wikidata and the Encyclopedia of Life. Each are geared towards different audiences and have varying levels of ease of use and maturity. I'll talk about the strengths and weaknesses of each project and how to join and query for data across spatiotemporal, governmental, cultural, and scientific domains using Linked Data query languages.

The advent of the Internet and heterogenous distributed software call for new tools for validating, documenting, coercing and transforming data. Surprisingly, conventional programming languages provide weak support for this because they strongly focus on behavior. Indeed, conventional type systems are designed towards fast code execution but only provides some type safety from a pure data perspective. In contrast, data requirements often require capturing precise data type definitions (e.g. the set of "integers less than 100", or the set {M,F}) but call for no behavior per se.

This talk is about Finitio (finitio.io), a language strongly biased towards data. Finitio has a dedicated type system for capturing, validating and coercing data, and an interoperability layer to play nice with existing programming languages. The talk is a guided tour to Finitio; it also briefly discusses its motivation and origin.

At the National Archives we have developed a simple Schema language for describing the data within a CSV file. This schema language which we call CSV Schema, enables you to make quite complex rule based assertions about the data within a CSV file. We have then built a tool (which we call CSV Validator) which will take this CSV Schema and a CSV file and validate it, producing a report of any problems.

Both the CSV Schema language (http://digital-preservation.github.io/csv-schema/csv-schema-1.0.html) and the CSV Validator tool are released freely as Open Source projects on GitHub. We believe that our tools will be of interest to others, such as those attending this conference.

When 2/3 of CSVs on data.gov.uk aren't easily machine-readable, we have a problem. We need new tools that help data publishers to make usable data, and help them maintain it over time. CSVlint.io is such a tool, that validates CSV files, suggests improvements, and checks against datapackage schemas. Come and learn how it can be used to make data publishing better.

I'm one of the founders of Nestoria - a search engine for residential real estate operating in nine markets and used by millions of people. We process (error check, de-duplicate, geocode, etc) over 10 million listings a day. given that purchasing a property is the largest financial transaction most people will ever face, you might think the industry would value high quality data. You would be very wrong. I'll share some of the challenges we've faced, the lessons we've learned, and hopefully provide a few entertaining examples of data gone bad.

How do Wikimedia project communities work? How do data hackers interface and interact with these communities? What is at stake and who are the stakeholders?

Join this talk to learn by example, through the story of the Open Access Signalling Project. This project's focus is to improve existing Wikipedia citations of Open Access research articles and other such academic works. This is one path among parallel initiatives (past and present) to improve how references work on Wikipedia, and across Wikimedia projects.

"A fact is only as reliable as the ability to source that fact, and the ability to weigh carefully that source." - WikiScholar proposal 'A free and universal bibliography for the world' (circa 2006 - 2010, status: closed)

In 2014, La Nacion in Argentina launched VozData, a website to crowdsource senate spendings by asking people to transcribe information from 6000 scanned PDF documents from the senate. This is a talk about the code that created that website and it can be used with any document set and any data you may need to take from them.

The ContentMine aims to liberate facts from the scientific literature through Natural Language Processing, Computer Vision and good old Regular Expressions. At present we are liberating chemistry, biodiversity, species, places, dates, not only from text but also diagrams (PDF, PNG, etc.) Everything is Free/Open (software, specifications and output) and we are working to build a bottom-up community where domain experts can build on our infrastructure.

We have built an online mapping service that lives and bleeds CSVs. CartoDB supports drag and drop import of CSVs and in seconds gives you interactive maps of your data. From there, you can share the maps or share links to export your data as new CSVs! It also gives you SQL access to your data, so you can filter and manipulate data and then provide those results as CSV exports! Getting data into CartoDB isn't limited to CSVs on your desktop, you can also point the service at files hosted online and tell it to sync your maps to that source file. This can give you a real-time map of data hosted on servers all around the world! Add on top of it the ability to georeference your CSVs by administrative areas, named places, IP addresses, and many other attributes, and you can turn almost any CSV into a mappable dataset. Here, I'm going to show you how CartoDB is built on and contributes to an ecosystem of CSVs.

We would all like for our applications to run off dat-powered real-time feeds. More often than not, however, the data we serve doesn't change that often - or our data apps are one-off presentations anyway.

In those cases, a dynamic server often isn't needed to power an application - they can just as well run off flat files and a cheap CDN. In this workshop I want to present some recipes and tools for making static file applications, but also have a discussion to learn from others and discuss some of the missing bits for making truly great flat-file apps.

Even within government agencies opening data is a hot issue. Confronted with functional silos and difficult access to another silos data, the belgian Ministry of Defence started building an internal open data platform powered by open source software and csv files in 2009.

This talk will provide a brief technical overview of the platform used to distribute the data (including the role of csv files), the volume of data we ship nightly, whether or not we killed the silos and what it's like to operate within a special kind of enterprise.

If data is fallible, can anything be known? Moreover, how are we going to get a reasonable result from this new dump and still have time for lunch? What is the difference between "cleaning" and "analysis"? Will epistemological theorizing improve our quotidian experiences working with data, or is it all just sesquipedalian nonsense? Does skeptical coding bring us closer to the truth?

Datacentral is a simple system we made for CentralDeDados.pt, an independent public data hub in Portugal. Making use of the Data Package standard, we wrote a proof-of-concept static site generator that pulls each CSV dataset from a Git repository and generates the site based on the contained metadata (using daily cron jobs to re-generate and update the site). The result is an easily deployable, replicable and complete HTML5 data portal site.

As part of mejoratuescuela.org, we opened public data that was given in PDF's and obstrusive websites. We made an study based on this data and we achieved public policy changes, we helped the open data movement and community in México. Its incredible the impact we made only by a team of 3 people and I want to share my side of the story.

In this talk I will demo a new R package, testdat, that provides a suite of functions that allow users to unit test tabular data, much like unit testing for code. Our package allows researchers to write expectations, as one would do with code, and quickly identify cryptic issues, especially when reading large numbers of files. We also describe the major functionality of testdat along with a few use-cases and related tools. (full abstract)

Openrosa has emerged as a popular standard for (mobile) data collection with a growing eco-system of applications and services. The proposed talk will focus on Airwolf, an Openrosa compatible tool developed for Text to Change, an organization that among others operates a small call center in Uganda. It will explain for what kind of data collection efforts a call center is the appropriate method and detail some of the challenges that Airwolf has to tackle. These include offline capabilities, anonymization of results and data export that is both useful for statisticians and readable for less technical people.

Real-world spreadsheets of data come in many formats and with widely varying headers; we need a language to navigate through them to locate the data which we need.

Inspired by XPath and building upon OKFN's Messytables, XYPath isn't quite that language, but is a working prototype to explore what it might look like.

A vege-table is a data table: each row of the table is an item in a collection, and each column is a property of those items. These properties are all JavaScript functions, so their values can be computed or fetched from remote resources, and merged from different sources. This talk will describe the project’s background - including limitations of current tools for building, manipulating and publishing data tables - and examples of its use with real data.

Many people consider spreadsheets to be data, but more often, they are actual programs. In this talk Felienne explains why and also provides help in how to deal with 'legacy' spreadsheets.

Open data does not mean that the data can easily be consumed. Every source has a different format that requires a dedicated parser. I've recently open sourced 2 tools (https://github.com/pauldeschacht/pdf2csv and https://github.com/pauldeschacht/paxparser) that help the extraction and standardization of diverse formats. I will briefly discuss these projects, but mostly I hope to build a community around such tools.

Open data isn’t a new concept, at least to those attending this conference. Now that more governments, agencies, and organizations are on board to release valuable data and information, what standards, guidelines, and other policy instruments need to be established in order to make that data useful to the public? Given that nations have similar data types, creating these policies with an international lens would create greater interoperability and lead to increased use. How do we get here? Who is the governance body? Come to this interactive presentation to share, learn, and move this conversation forward!