Article written by Martin Humenberger
Visual localization, which estimates the position and orientation of a camera in a map based on query images, and structure from motion (SFM), which is one of the most popular ways to build this visual localization map, are fundamental components in the development of technologies such as autonomous robots and self-driving vehicles.
However, a major barrier to research in visual localization and SfM lies in the format of the data itself. Although many good public datasets for evaluation now exist, these datasets are all structured using different formats. As a result, data importers and exporters must often be modified and coordinate systems or camera parameters almost always must be transformed. Even more data conversion is necessary if you want to use a combination of multiple tools in a single pipeline. Many times, existing data formats don’t include the types of data needed for a specific application, especially when it comes to data such as WiFi or other non-image sensors.
To overcome this research barrier, we created the kapture format. Kapture includes a wide range of data to store and comes with several format converters and data processing tools. On the one hand, it will make public datasets easier to use and on the other hand, it will allow processed data (such as local or global features or matches) to be shared easily (again in one common format). By sharing our kapture format with the research community, our aim is to facilitate future research and development in topics such as visual localization, SfM, VSLAM and sensor fusion.
The release contains the following main features:
Given a known 3D space representation, visual localization is the problem of estimating the position and orientation of a camera using query images. Structure from Motion (SfM) is one of the most popular ways to reconstruct a 3D scene from an unordered set of images.
Kapture can be used to store many types of data collected for visual localization and SfM. Examples include:
It can also store data computed during various stages of the process, such as
Kapture also includes a set of Python tools to load, save, and convert datasets to and from kapture. General formats such as COLMAP, openmvg, OpenSfM, bundler, image_folder, image_list, and nvm, as well as a few formats specific to particular datasets such as IDL_dataset_cvpr17, RobotCar_Seasons, ROSbag cameras+trajectory, SILDa, and virtual_gallery are all supported.
Our example pipelines take you through the process of localizing query images on a map, which consists of two major parts: i) building the map and ii) localizing a query. In each pipeline, we first show you how to build the map using SfM and known poses, and then we show you how to localize query images. We also explain how to use kapture to evaluate the precision of the obtained localization against the ground truth.
We’ve already converted a number of datasets to the kapture format. For instance, kapture can be used to process all the datasets included in ECCV 2020’s Visual Localization Challenge (the Aachen Day-Night, Inloc, RobotCar Seasons, Extended CMU-Seasons, and SILDa Weather and Time of Day datasets). The images in these datasets are meant to include the many challenging scenarios that arise in real driving situations and include changes in the time of day, season of the year, and outdated reference representations. The datasets also include images with occlusion, motion blur, extreme viewpoint changes, and low-texture areas.
If you already have your SfM or visual localization processing tools up and running, you just need to integrate kapture support once, after which you can use all the datasets without any additional conversion or glue code writing. Additional details and how to obtain an updated list of datasets can be found in the kapture tutorial.
If you find kapture useful, we encourage contributions!
You are welcome to provide your own dataset in kapture format (we’re happy to help), write new data converters, report bugs and suggest improvements, provide processed data (e.g., extracted features or matches) in kapture format, and add support for additional data types.
This article was first published on the blog of NAVER LABS Europe.
More information, news, and updates can be found on this website.
GitHub Repository: https://github.com/naver/kapture
License: BSD 3-Clause
Paper: M Humenberger, Y Cabon, N Guerin, J Morat, J Revaud, P Rerole, N Pion, C de Souza, V Leroy, and G Csurka: Robust Image Retrieval-based Visual Localization using kapture