We present our latest work on the CrowdTruth framework, titled “Human Computing for the Real World”, at the ICT Open 2017 conference on 21st and 22nd of March 2017. I made a new video that demonstrates the different aspects of the framework for dealing with ambiguity in data, crowdsourcing of human interpretations, and evaluating disagreement between annotations.
Yesterday at the Computable Awards the Vrije Universiteit, University of Amsterdam and IBM won the prize for “ICT project of the year in education” with the Watson Innovation Course. Furthermore, the project was highest rated across all nominees of all prize categories. The course is ongoing at the moment for the second time, with an improved setup and new state of the art tools for the students.
The course is run by Lora Aroyo, Anca Dumitrache, Benjamin Timmermans and Oana Inel from the VU, and Robert-Jan Sips and Zoltan Szlavik from IBM. In the course the students were challenged by Amsterdam Marketing to solve the issue of the increasing overcrowdedness of tourists in the city center of Amsterdam. The city is culturally rich with many places to visit, yet most visitors cluster around a limited set of popular locations. The students came up with ideas to motivate visitors to spread in the city and provide them with relevant information for their visit.
Brainstem tumors are a rare form of childhood cancer for which there is currently no cure. The Semmy Foundation aims to increase the survival of children with this type of cancer by supporting scientific research. The Center for Advanced Studies at IBM Netherlands is supporting this research by developing a cognitive system that allows doctors and researchers to quicker analyse MRI-scans and better detect anomalies in the brainstem.
In order to gather training data, a crowdsourcing event was held at the festival Lowlands, which is a 3-day music festival that took place from 19-21 August 2016 and welcomed 55k visitors. At the science fair, IBM had a booth that hosted both this research and showcase of the Weather stations of the Tahmo project with TU Delft.
In the crowdsourcing task, the participants were asked to draw the shape of the brainstem and tumor in an MRI scan. Gathering data on whether a particular layer of a scan contains the brainstem and determining its size should allow a classifier to recognize the tumors. Furthermore, the annotator quality can be measured with the CrowdTruth methodology by analysing the precision of the edges that were drawn in relation to their alcohol and drug use that we collected. The hypothesis is that people under influence can still make valuable contributions, but that these are of lower quality than sober people. This may make the reliability of online crowd workers more clear, because it is unknown under what conditions they make their annotations.
The initial results in the heatmap of drawn pixels give an indication of the overall location of the brainstem, but further analysis will follow on the individual scans in order to measure the worker quality and generating 3d models.
From 2 to 16th of July we organized the Big Data in Society Summerschool at the Vrije Universiteit Amsterdam. As part of our Collaborative Innovation Center with IBM, we presented an introduction of the technical and theoretical underpinnings of IBM Watson and discussed the use of big data and implications for society. We looked at examples of how the original Watson system can be adapted to new domains and tasks, and presented the CrowdTruth approach for gathering training and evaluation data in this context. The participating students, which ranged from bachelor to PhD level, said they learned a lot from the lectures and found the practical hands-on sessions very useful.
On Friday 22 January Gerard Smit (CTO for IBM Belgium, Netherlands, Luxembourg) and Prof. Hubertus Irth (Vice Dean and Research Director of the Vrije Universiteit Faculty of Earth and Life Sciences and Faculty of Sciences) officially launched the collaboration between the IBM Collaborative Innovation Center (CIC) and the VU Faculty of Sciences. In the event students of the Watson Innovation course pitched their projects to a mixed crowd of students, scientists, engineers and business clients.
In the Watson Innovation course, students used Watson to answer questions about Amsterdam, for which Amsterdam Marketing provided the data and use case. The app LocalBuddy was selected as winner, and the students received a prize for their achievements by Amsterdam Marketing.