home
vitrivr Logo
GSoC Logo

What is GSoC?

Google Summer of Code is designed to encourage student participation in open source development. Over the past 13 years, Google Summer of Code has brought together over 13,000 students and 608 open source projects to create over 33 millions of lines of code. Google will accept applications from student applications from March 12 to 27, 2018.

g.co/gsoc

vitrivr at GSoC

In GSoC 2018, we want you to work with all components of vitrivr. Cineast and ADAMpro work jointly to provide the user with a sketch-based multimedia retrieval engine which can be interacted with via Vitrivr-NG. Watch the following video to see a demo of our running system. To see how you can set up the vitrivr stack on your system, visit our getting started page.

This is vitrivr’s second participation to Google Summer of Code. Visit our GSoC 2016 page to learn more about our participation to GSoC 2016.

Student Application

To apply for one of our projects, we ask you for the following information:

Problem Statement
State very briefly the problem you are trying to solve, the goal of the project and the expected outcome. Be specific and realistic about the outcome and about what you want to achieve in three months.
Methodology
State the steps you think are required for solving the problem. Be specific, choose small doable parts that you can oversee and of which you can give good time estimate.
Project Plan
We want to see a detailed weekly project plan with well-defined milestones and deliverables (and potential breaks or absences).

In addition, we ask you to submit code to one of the following pre-tasks. For each of the tasks you are asked to write a piece of source code in your language of choice, in good style, no longer than 150 lines. Upload your code snippet anywhere and send the link to it to with subject “GSoC 2018 - Task X” (where X denotes the task you have solved). In case you have not yet submitted your application via the Google Summer of Code website, please provide a draft of your application (via link or attachment) in the email together with the solution of your chosen task.

It is important that you know well about vitrivr and about potential things to improve. Get in contact with the mentors to pick a good project that fits well to vitrivr and to you.

Ideas

Below, you can find some of our ideas on the directions in which we could push vitrivr together. Please consider them as starting points for your proposal. Of course, if you have other ideas, we would be very happy to hear them. Be creative and send us proposals that extend the ideas below – or that are advancing the vitrivr multimedia retrieval stack in another way. Feel free to contact us and get feedback on what you plan to do beforehand.

Task Software Details Technologies Principal Mentor Difficulty
vitrivr App Cineast vitrivr comes with a web UI which has the advantage that it can be accessed by a large variety of devices. However, as a disadvantage, the UI cannot make use of any possibly further available native functionality, such as touch input, a camera, GPS sensors for linking photos or videos to the real world, etc. Furthermore, it is not optimized for small screens. In this work, the goal is to implement a native Android-based vitrivr app that can be used on an Android phone or tablet. The app could also make use of the spatial retrieval capabilities of vitrivr to limit results to only those which are somehow relevant to your current location. This project can be done in collaboration with the Red Hen project. Java, Kotlin, etc. TBD
medium
[email protected] vitrivr While vitrivr is currently capable of processing the vast majority of media files commonly found in home media collections, its usability in these situations could still be vastly better. The aim of this project would be to make it easier to deploy vitrivr at home and use it to manage a home media collection. In order to do this, one could for example increase the capabilities of the UI in order to allow collection management or build plugins for other media management tools such as Plex to enable them to leverage the search capabilities of vitrivr. Java and others TBD
medium
bring cthulhu back cthulhu The distributed extraction management tool cthulhu which was created during Google Summer of Code 2016 sadly did not keep pace with the development of the other components of vitrivr and is therefore currently incompatible with the latest version of vitrivr. The aim of this project would be to bring cthulhu back to a point where it can communicate with vitrivr and add new functionality to it. Java TBD
medium
vitrivr and iiif vitrivr The international image interoperability framework describes a standard to interact with images which is being adopted by an increasing number of museums, archives and other institutions which store large collections of digital images. The aim of this project would be to make vitrivr compatible with the apis specified by iiif so it can be used to interact with any iiif capable data sources easily. Java, web TBD
hard
vitrivr and TensorFlow Cineast The staggering amount of progress in the field of deep learning bares many fruits, some of which could be beneficial for vitrivr as well. While there is a way to use TensorFlow in vitrivr, it uses an unofficial and outdated interface which makes it difficult to employ cutting edge models. The aim of this project would be to replace the current TensorFlow interface with the official one for Java and build the necessary structures for it to be able to use current TF-models. Java, TensorFlow TBD
medium
vitrivr in the cloud vitrivr While vitrivr can be and has been deployed in a cloud environment, it involves manual changes every time the scaling is to be changed. The aim of this project would be to make vitrivr cloud-aware by adding for example dynamic load balancing capabilities which would facilitate scale-out. Adding the possibility to feed vitrivrs vitals into for example a Grafana dashboard would also go in this direction. Java TBD
hard
Subtitle support vitrivr vitrivr currently supports a few clear-text formats of subtitles but there are many which are not supported yet. The aim of this project would be to add support for various subtitle formats to Cineast and Vitrivr-NG and maybe even build an interface to CCExtractor. This project will explore synergies from combining CCExtractor and vitrivr and will thus be done in collaboration of both organizations. Java, web, etc. TBD
medium
vitrivr and structured metadata vitrivr Add support for structured metadata on objects and the search within these structured metadata on the UI. Java, web TBD
easy
Art from Data vitrivr vitrivr does have some capabilities for feature visualization which are however currently not accessible via the APIs. The aim of this project would be to first re-enable the access to these visualization capabilities and subsequently expand among them. This project might also apply the visualization features to the large newscape video collection, in a collaboration between vitrivr and Red Hen. Java, web, etc. TBD
medium
vitrivr the explorer vitrivr While vitrivr works great if you know what you are looking for, its capacity for exploration is somewhat limited. The aim of this project would be to expand the browsing and exploration capabilities of vitrivr, for instance by offering visualizations leveraging semantic topologies. This can be done in collaboration with the Red Hen organization, on the basis of the newscape video collections and the semantic annotations. Java, web, etc. TBD
medium
Enhanced 3D retrieval Cineast vitrivr has the capabilities to search for 3d models based on 3d sketches or 3d examples. It currently lacks the capability to search 3d objects which can be seen in pictures or videos. It’s currently also not possible to search for videos or images in which a certain 3d model might be visible. The aim of this project would be to explore ways in which this gap between 3d and visual objects can be bridged. Java, etc. TBD
hard
Speech and Text from video vitrivr It has been proven useful when searching for video segments to be able to search using dialog or text which is shown on screen. The aim of this project would be to integrate state of the art OCR and Speech-to-Text methods into vitrivr in order to support such queries. Java, etc. TBD
medium
vitrivr the organizer vitrivr Especially in the context of video, vitrivr assumes that every video file contains only one piece of content. But what if this is not the case? In instances where a video contains several pieces of content, for example in the case of continuous television recordings, uncut film footage, etc., vitrivr is not currently able to reorganize this content into a more semantically meaningful form. The goal of this project would be to add such segmentation and organization capabilities – both manual and automatic – to vitrivr. This can be applied on a real world dataset for the detection of show boundaries in TV recordings using the newscape collection of Red Hen in a collaboration of both organizations. Java, etc. TBD
medium

Mentors

This list has the contact information of some of our mentors. It will be updated over time.

Mentor Profile Contact Languages
Heiko Schuldt http://dbis.dmi.unibas.ch/team/heiko-schuldt English, German
Ivan Giangreco http://dbis.dmi.unibas.ch/team/ivan-giangreco
English, German
Luca Rossetto http://dbis.dmi.unibas.ch/team/luca-rossetto English, German
Ralph Gasser http://dbis.dmi.unibas.ch/team/ralph-gasser English, German
Silvan Heller http://dbis.dmi.unibas.ch/team/silvan-heller English, German
Mark J. Williams http://film-media.dartmouth.edu/people/mark-j-williams English
Francis Steen http://commstudies.ucla.edu/content/francis-steen-phd English
Carlos Fernandez English

All mentors are available on Google Hangout and skype.

Contact

Please feel free to contact us if you have any questions. To make it easier for us to organize our emails, please add “GSoC 2018” in your subject line. You can contact us via our email address or you can contact the mentors directly.

Mailing List

We have set up a mailing list for the vitrivr project. You can use the mailing list for questions regarding the code, bugs, etc. Note that the mailing list is publicly visible.

chevron_rightMailing List