We present our latest work on the CrowdTruth framework, titled “Human Computing for the Real World”, at the ICT Open 2017 conference on 21st and 22nd of March 2017. I made a new video that demonstrates the different aspects of the framework for dealing with ambiguity in data, crowdsourcing of human interpretations, and evaluating disagreement between annotations.
Our demo of ControCurator titled “ControCurator: Human-Machine Framework For Identifying Controversy” will be shown at ICT Open 2017. In this demo the ControCurator human-machine framework for identifying controversy in multimodal data is shown. The goal of ControCurator is to enable modern information access systems to discover and understand controversial topics and events by bringing together crowds and machines in a joint active learning workflow for the creation of adequate training data. This active learning workflow allows a user to identify and understand controversy in ongoing issues, regardless of whether there is existing knowledge on the topic.
Yesterday at the Computable Awards the Vrije Universiteit, University of Amsterdam and IBM won the prize for “ICT project of the year in education” with the Watson Innovation Course. Furthermore, the project was highest rated across all nominees of all prize categories. The course is ongoing at the moment for the second time, with an improved setup and new state of the art tools for the students.
The course is run by Lora Aroyo, Anca Dumitrache, Benjamin Timmermans and Oana Inel from the VU, and Robert-Jan Sips and Zoltan Szlavik from IBM. In the course the students were challenged by Amsterdam Marketing to solve the issue of the increasing overcrowdedness of tourists in the city center of Amsterdam. The city is culturally rich with many places to visit, yet most visitors cluster around a limited set of popular locations. The students came up with ideas to motivate visitors to spread in the city and provide them with relevant information for their visit.
Today I gave a talk in our Weekly AI meeting on the topic of ControCurator. This is a project that I am currently working on, which has the goal to enable the discovery and understanding of controversial issues and events by combining human-machine active learning workflows.
In the talk I explained the issue of defining the space of a controversy, and how this relates to for instance wicked problems. You can see the slides below.
Brainstem tumors are a rare form of childhood cancer for which there is currently no cure. The Semmy Foundation aims to increase the survival of children with this type of cancer by supporting scientific research. The Center for Advanced Studies at IBM Netherlands is supporting this research by developing a cognitive system that allows doctors and researchers to quicker analyse MRI-scans and better detect anomalies in the brainstem.
In order to gather training data, a crowdsourcing event was held at the festival Lowlands, which is a 3-day music festival that took place from 19-21 August 2016 and welcomed 55k visitors. At the science fair, IBM had a booth that hosted both this research and showcase of the Weather stations of the Tahmo project with TU Delft.
In the crowdsourcing task, the participants were asked to draw the shape of the brainstem and tumor in an MRI scan. Gathering data on whether a particular layer of a scan contains the brainstem and determining its size should allow a classifier to recognize the tumors. Furthermore, the annotator quality can be measured with the CrowdTruth methodology by analysing the precision of the edges that were drawn in relation to their alcohol and drug use that we collected. The hypothesis is that people under influence can still make valuable contributions, but that these are of lower quality than sober people. This may make the reliability of online crowd workers more clear, because it is unknown under what conditions they make their annotations.
The initial results in the heatmap of drawn pixels give an indication of the overall location of the brainstem, but further analysis will follow on the individual scans in order to measure the worker quality and generating 3d models.
From 2 to 16th of July we organized the Big Data in Society Summerschool at the Vrije Universiteit Amsterdam. As part of our Collaborative Innovation Center with IBM, we presented an introduction of the technical and theoretical underpinnings of IBM Watson and discussed the use of big data and implications for society. We looked at examples of how the original Watson system can be adapted to new domains and tasks, and presented the CrowdTruth approach for gathering training and evaluation data in this context. The participating students, which ranged from bachelor to PhD level, said they learned a lot from the lectures and found the practical hands-on sessions very useful.
On the 22nd of March we presented our latest work on CrowdTruth at the ICT.OPEN 2016 conference. We are happy to announce that our poster received the best poster award in the Human and the Machine track. Furthermore, Anca Dumitrache gave a presentation and pitched our poster which resulted in the 2nd prize for best poster of the conference. It is a good signal that from the almost 200 posters the importance of the CrowdTruth initiative was recognized.
Today I visited the Guide to the network society conference at the Vrije Universiteit Amsterdam. I attended a session on the digitization of cities, in which it became clear to me that cities are going to a major change. Due to digital communication and e-commerce, the need for a main city center with shops and facilities like banks has decreased drastically. Because of this, the way we look at urban development is changing.
An interesting talk was given by Eduardo Diaz on the work of GeoCraft. In this research minecraft is used as a tool for children to gather ideas on how do develop their new neighborhood. This can help the municipalities and government agencies better develop urban areas. I see this as a sort of gamified crowdsourcing, with the interesting aspect that rather than developing their own platform they used a game that many children already play. Using an existing platform can be a valuable approach for our crowdsourcing research. Though, limitations can arise with the functionality and design of such platform. Nevertheless, there are many games and forums where enthusiasts share their knowledge to form a sort collective intelligence. I believe this can be a valuable source of information for government agencies, which will become more important in the future.
The second session was about the success and doom scenarios of intelligent computers. Guszti Eiben, Zoltán Szlávik and Noel Sharkey each presented both their hypothetical success story and the worst case outcome of the recent developments in artificial intelligence and robotics. Guszti presented his view on the possibility of killer robots and the ethical responsibilities we have as researchers. From his work on evolutionary computing, he learned it is essential that in the future we allow robots to reproduce only in a centralized system. This allows the evolution of robotics to be stopped by disabling the central replication, in the case it were ever to become hostile to humanity. Zoltan made an interesting statement that it is not intelligent robots that threaten the future of humanity, but it is humans that will try to abuse power with these robots. Nevertheless his experience is that robots are already have a positive contribution to society, for instance in education and helping the elderly.
On Friday 22 January Gerard Smit (CTO for IBM Belgium, Netherlands, Luxembourg) and Prof. Hubertus Irth (Vice Dean and Research Director of the Vrije Universiteit Faculty of Earth and Life Sciences and Faculty of Sciences) officially launched the collaboration between the IBM Collaborative Innovation Center (CIC) and the VU Faculty of Sciences. In the event students of the Watson Innovation course pitched their projects to a mixed crowd of students, scientists, engineers and business clients.
In the Watson Innovation course, students used Watson to answer questions about Amsterdam, for which Amsterdam Marketing provided the data and use case. The app LocalBuddy was selected as winner, and the students received a prize for their achievements by Amsterdam Marketing.
Today the students of the first Watson Innovation course by the Vrije Universiteit Amsterdam and IBM Netherlands presented their group work at the VU Intertain Lab. Representatives from Amsterdam Marketing and IBM Netherlands were present to evaluate the ideas, applications and business plans of the groups. The groups have been working on their Watson powered apps since last November, using the Watson Engagement Advisor and IBM Bluemix. The most interesting project groups will be selected to present their work again next Friday at IBM.