PRESS RELEASE

You want to take a nice picture in the nature with your mobile phone. You point the camera at the landscape and the device will offer you the names of the surrounding hills, their height, hiking paths or even contours directly on the screen in augmented reality. Thanks to the fact that the new software tool of researchers from CPhoto@FIT BUT knows where the photographer stood and what he/she photographed, or what the place looks like in various conditions, then at home you can edit the photo in various ways, for example sharpening or changing shadows. Or transfer to the snapshot location again via virtual reality.

All this is done by new software developed by a team from the research group of computer photography of the Faculty of Information Technology of the Brno University of Technology in cooperation with Adobe Research. They introduced the new tool together at the prestigious ECCV conference, now they are waiting for a patent.

"Our software can refine the position and orientation of the camera in the outdoor environment. In the mobile application, it will offer various information about the surroundings through augmented reality - river and mountain names, contours or distance to the mountain hut - it can simply display any topographic information in real terrain." the head of the CPhoto@FIT research group Martin Čadík describes.

Thanks to GPS positioning, the mobile phone displays a synthetic view of the landscape, similar to, for example, Google Earth. From the photo or screen, it then detects significant points, such as the outlines of hills, rivers or forests, compares them with terrain models and thus can determine the position and orientation of the camera to meters accurately. The comparison of points from a photograph and from a 3D terrain model is done automatically using a neural network. Researchers have trained this in thousands of landscape photographs from their own archives and images downloaded from the Internet.

"We used to use these images just to compare individual points, so we compared photos with photos. But there were a number of disadvantages - we couldn't make localization from where we didn't have images. We now compare photos directly with 3D terrain models. They cover the whole planet, therefore, the places where people do not go, also they contain data with textures from different seasons, which helps to locate if the landscape changes," Martin Čadík explains. In the world of computational photography, this is a major step that has been made possible by advances in the development of neural networks and the availability of accurate field models with textures.

The algorithms then help users especially at home at the computer. Thanks to the software, it already knows perfectly the place where the picture was taken, as well as the orientation of the camera, i.e. the place that the photographer wanted to immortalize. This makes it possible to edit images that would otherwise be very complicated. It allows users to focus on another peak, add shadows, or change the lighting of a photo. The software can also plant the captured photo directly into the field and can then return the photographer directly to the place where the image was taken in virtual reality. Thanks to special glasses, the photographer can, for example, virtually show his/her friends or relatives the place where he/she took the pictures and what the surroundings look like outside the photo.

The tool was created within the project of the Ministry of Education, Youth and Sports of the Czech Republic Topographic image analysis using deep learning methods. FIT scientists want to build on it with further research, which should be able to locate the place and orientation of the camera on a larger scale thanks to a neural network and field models, even without a prior rough estimate of the position from GPS.