vitrivr Logo
GSoC Logo

What is GSoC?

Google Summer of Code is designed to encourage student participation in open source development. Over the past 16 years, Google Summer of Code has brought together over 16,000 students and 715 open source projects to create over 33 millions of lines of code. Google will accept applications from student applications from March 29 to April 13, 2021.

g.co/gsoc

vitrivr at GSoC

As in previous years, we want you to work with all components of vitrivr. Cineast and Cottontail DB work jointly to provide the user with a sketch-based multimedia retrieval engine which can be interacted with via Vitrivr-NG. Watch the following video to see a demo of our running system. To see how you can set up the vitrivr stack on your system, visit our getting started page.

This is vitrivr’s third participation to Google Summer of Code. Visit our GSoC 2016 or GSoC 2018 pages to learn more about our previous participations.

Student Application

To apply for one of our projects, we ask the applicants to write a brief project proposal document (Google Doc, Word etc.), containing the following information:

Problem Statement
State very briefly the problem you are trying to solve, the goal of the project and the expected outcome. Where applicable, make an explicit reference to one of the projects found on this website. Be specific and realistic about the outcome and about what you want to achieve in three months.
Methodology
State the steps you think are required for solving the problem and connect them to the components in the vitrivr stack where applicable (i.e. vitrivr NG, Cineast and Cottontail DB). Be specific, choose small doable parts that you can oversee and of which you can give good time estimate.
Project Plan
We would like to see a detailed weekly project plan with well-defined milestones and deliverables (and potential breaks or absences).

In addition, we require all applicants to setup the vitrivr stack on their local machine prior to handing in an application. Hence, before sending the application, please carry out the following steps:

  1. Checkout and setup the necessary components (i.e. Cottontail DB, Cineast, vitrivr NG. Please mind that for, e.g., Cottontail DB, there is also a Docker image.
  2. Run an extraction on a small video collection (at least one file) on your local machine. Mind that there is an example repository with extraction configuration files (see link).
  3. Make a screenshot or video of you retrieving an item from the extracted collection using vitrivr NG (the UI).

As a proof-of-work, we require you to send us the video or image of you using vitrivr with your project application. You can send the proof-of-work to [email protected]

It is important that you know well about vitrivr and about potential things to improve. Get in contact with the mentors to pick a good project that fits well to vitrivr and to you.

Ideas

Below, you can find some of our ideas on the directions in which we could push vitrivr together. Please consider them as starting points for your proposal. Of course, if you have other ideas, we would be very happy to hear them. Be creative and send us proposals that extend the ideas below – or that are advancing the vitrivr multimedia retrieval stack in another way. Feel free to contact us and get feedback on what you plan to do beforehand.

Project Involves Details Technologies Principal Mentor Difficulty
Multimedia Collection Management vitrivr NG, Cineast Currently, vitrivr offers no support for managing items in a media collection. This rather easy project extends the existing vitrivr-ng UI and Cineast with a new endpoint supporting create, read, update and delete (CRUD) capabilities for media. UI should also be expanded to allow for listing the media collection, enable exploration thereof in terms metadata etc. Java, Typescript Florian, Silvan
easy
Vizard: Installer All This is a subproject to Vizard, a dedicated web UI that should offer administrative functionality for the vitrivr stack. The aim of this project is to provide a simplified installation and deployment of vitrivr as an app for mac, windows (and maybe linux) via a dedicated web UI. The application should take care of checking out the components of their respective repositories, compiling, installing and bootstrapping them with a basic configuration. Java, Kotlin Loris, Ralph, Florian
easy
Vizard: Extraction UI Cineast This is a subproject to Vizard, a dedicated web UI that should offer administrative functionality for the vitrivr stack. Media extraction in Cineast is currently managed via JSON-Files, which is not very user-friendly. The aim of this project is to extend the Vizard UI with capabilities to configure extractions and start media extraction for a specific collection. Java, Kotlin Loris, Ralph, Florian
medium
Vizard: Configuration Manager vitrivr NG, Cineast This is a subproject to Vizard, a dedicated web UI that should offer administrative functionality for the vitrivr stack. Both vitrivr-ng and Cineast have a variety of config options which are not easily understood by new users and contributors. This project can be done in a shorter or longer version. The shorter version simply provides an admin UI where config options can be changed and the user has to reload vitrivr-ng or restart Cineast. In the longer project, the changes should be applied to running instances of Cineast dynamically and with minimal interruptions. Java, Kotlin Loris, Ralph, Florian
easy
vitrivr reads Cineast Text in a video often conveys information which is not easily expressed otherwise. This project deals with the integration of state of the art scene-text transcription into a dedicated feature module in Cineast. Effort of the project depends on whether the feature module uses an existing python library or if the model is implemented properly in Java via TensorFlow bindings (which would be the preferred way to go). Java, TensorFlow Mahnaz, Luca, Silvan
hard
vitrivr listens Cineast Spoken words are a very salient and easily remembered part of any video and therefore interesting for search. The idea behind this project is to integrate state of the art speech transcription methods such as DeepSpeech into vitrivr to make video transcription an inherent part of the retrieval pipeline. Java, TensorFlow Mahnaz, Silvan
hard
IIIF read Cineast The International Image Interoperability Framework (IIIF) describes a standard to interact with images which is being adopted by an increasing number of museums, archives and other institutions that store large collections of digital images. The aim of this project is to make vitrivr compatible with the APIs specified by IIIF so that it can be used to interact with any IIIF capable data sources easily. Java Luca, Florian
medium
IIIF endpoint Cineast The International Image Interoperability Framework (IIIF) describes a standard to interact with images which is being adopted by an increasing number of museums, archives and other institutions which store large collections of digital images. The aim of this project would be to make cineast provide a IIIF endpoint to use the IIIF apis for search and lookup. Java Luca, Florian
medium
Cottontail DB UI Cottontail DB Cottontail DB is the storage engine for vitrivr and currently used to execute all types of Boolean and nearest neighbor search (NNS) queries. Apart from a CLI, it currently offers very little user-facing interfaces to manage and query the data it contains.
This project thus deals with building a web-based UI for Cottontail DB. For starters, the UI should offer basic management functionality for schemas, entities and indexes as well as basic administrative functions. In a second step, one could also add the ability to inspect data and even simple data population functionality (e.g. import and random generation).
Kotlin Ralph
medium
Cottontail DB goes ScaNN Cottontail DB Cottontail DB is the storage engine for vitrivr and currently used to execute all types of Boolean and nearest neighbor search (NNS) queries. To speed-up NNS, Cottontail DB supports a wide range of index structures. This project would focus on adding another such index based on ScaNN to Cottontail DBs. Kotlin Ralph
medium
vitrivr goes Polypheny Cineast With the recent integration of Cottontail DB into Polypheny DB, the polyglot database system has gained the ability to execute nearest neighbor search (NNS) with very competitive execution times. Being able to leverage other storage engines as well, such as HSQLDB, Polypheny DB is able outperform a pure Cottontail DB-based solution, since it can choose the appropriate storage engine based on the type of workload expected. This project will deal with adapting Cineast to enable it to use the JDBC/SQL-Interface provided by Polypheny to execute the different types of queries it uses. In addition to simply connecting the two systems, however, this project should investigate how the query model of Cineast can profit from Polypheny DBs capabilities and what types of operations can be offloaded to the database, for a more efficient and performant execution. Java Ralph, Silvan
hard
vitrivr explores vitrivr While vitrivr works great if you know what you are looking for, its capacity for exploration is somewhat limited. The aim of this project would be to expand the browsing and exploration capabilities of vitrivr, for instance by offering visualizations leveraging semantic topologies possibly based on Machine Learning. This can be done in collaboration with the Red Hen organization, on the basis of the newscape video collections and the semantic annotations. Java, etc. Luca, Ralph
easy
Gesture Retrieval for Annotation vitrivr Especially in the context of video, vitrivr assumes that every video file contains only one piece of content. But what if this is not the case? In instances where a video contains several pieces of content, for example in the case of continuous television recordings, uncut film footage, etc., vitrivr is not currently able to reorganize this content into a more semantically meaningful form. The goal of this project would be to add such segmentation and organization capabilities – both manual and automatic – to vitrivr. This can be applied on a real world dataset for the detection of show boundaries in TV recordings using the newscape collection of Red Hen in a collaboration of both organizations. Java, etc. Mahnaz, Ralph
medium
vitrivr watches vitrivr NG One of the few search modalities vitrivr does not yet support is query-by-example-video. There are various ways to implement this, e.g., by using gestures detected in a video, semantic concepts (i.e. using machine learning), motion or visual similarity. For this, a user friendly UI is needed that accepts both video as well as webcam input, streams it to Cineast for querying, displays the results to the user and possibly accepts feedback from the user. The existing UI already supports some of this capability, so the project can both build upon that or design a new UI. Typescript, Java Luca, Mahnaz
hard
Temporal Localization and Gesture Detection Cineast An important step in gesture recognition is to detect gestures and to localize them in a temporal dimension. Temporal Segmentation Networks (TSN) are one of the methods used for segmenting and localizing activities in videos. With a little modification, this method can be used to localize and segment multiple gestures in one video. This can be done in collaboration with the Red Hen organization, on the basis of the newscape video collections and the semantic annotations. Java Mahnaz, Silvan
high
Gesture Compatible UI in vitrivr vitrivr NG One of the important components for the gesture retrieval in vitrivr is a query by gesture compatible UI. This functionality was partly developed in previous GSoC projects with pose cues. The successful implementation should support limited length videos recorded by the webcam and forward it to the cineast for feature extraction. Typescript Mahnaz, Silvan
high

Mentors

This list has the contact information of some of our mentors. It will be updated over time.

Mentor Profile Contact Languages
Heiko Schuldt http://dbis.dmi.unibas.ch/team/heiko-schuldt/ English, German
Ralph Gasser http://dbis.dmi.unibas.ch/team/ralph-gasser/ English, German
Silvan Heller http://dbis.dmi.unibas.ch/team/silvan-heller/ English, German
Mahnaz Parian-Scherb https://dbis.dmi.unibas.ch/team/mahnaz-parian-scherb/ English
Luca Rossetto https://www.ifi.uzh.ch/en/ddis/people/rossetto.html English, German
Loris Sauter https://dbis.dmi.unibas.ch/team/loris-sauter/ English, German
Florian Spiess https://dbis.dmi.unibas.ch/team/florian-spiess/
English, German

All mentors are available on Google Hangout, Skype, Zoom or Slack.

Contact

Please feel free to contact us if you have any questions. To make it easier for us to organize our emails, please add “GSoC 2021” in your subject line. You can contact us via our email address or you can contact the mentors directly.

chevron_rightvitrivr @ Slack

Mailing List

The old mailing list has been discontinued in favor of GitHub discussions. You can use that forum for questions regarding the code, bugs, etc. And of course we’re always happy to get issues and pull requests!

chevron_rightDiscussions: Cineast chevron_rightDiscussions: vitrivr NG chevron_rightDiscussions: Cottontail DB