Utilizing XR technology and digital twins for real-time geo-registration

Geo-registration is the process of assigning 3D-world coordinates to the pixels of an image. The purpose of this is usually to build offline maps. 

The raw metadate used during this process is not completely accurate. Map accuracy varies between regions, GPS has built-in errors and typically has a low update rate, angular direction sensors are prone to BIAS errors, etc. 

For precise geo-registration, image processing algorithms must be used to match an image with an orthophoto (aerial photograph or satellite imagery geometrically corrected such that the scale is uniform).

Orthographic views project at a right angle to the data plane. Perspective views project from the surface onto the datum plane from a fixed location.

Geo-registration using image processing is traditionally done as follows:

1. Lens distortion corrections
2. Orthorectification of the image to correct an orthogonal perspective with respect to the ellipsoid
3. Image matching algorithm

Block diagram of a classic geo-registration process

Oversight geo-register real-time video streams by utilizing a physical engine and environmental digital twins.

What is it good for? 

As an example, let's look at a drone operator. An operator usually has two screens, one for video and one for map.

Using the video feed, the operator continuously tries to comprehend what he sees. Without prior experience with the scene, it is almost impossible for him to understand what he sees.

Additionally, aerial cameras typically have a narrow Field-of-View, making this task even more challenging. 

Video alone is not enough for a drone operator to understand what he sees!

Screen shot taken from a DJI drone

Different sensors (resolution, lens, amount of light, etc.) and different environmental conditions (lighting conditions, weather, etc.) pose two major challenges in matching an image from different sources.

Oversight intends to perform geo-registration directly, meaning that the video or the camera will not be corrected, but an identical camera will be created in a physical engine that produces the map in real time.

Simulating the environment where the video was taken using the physical engine by taking into consideration lighting conditions (time of day, location of the sun, shadows) and weather conditions (visibility, cloudiness) will offer agnostic approach to solve this challenge.

This will allow images to be matched instantly with different image processing algorithms, such as template matching.

In addition, Oversight provides VR or AR environments that mimic the human brain's understanding of the environment.  

Oversight uses an XR headset to give the operator much more natural control over the aerial vehicle by using head and eye movements.

Oversight demonstration of geo-registration at the US AWE expo in June 2022