MIT researchers present a new computer vision system that turns any shiny object into a camera of sorts: enabling the observer to see around corners or behind obstacles.

Valuable and often hidden information about a person’s immediate surroundings can be gained from the object’s reflection. By repurposing them as cameras, one can perform previously unimaginable feats, such as looking through walls or into the sky. This is challenging because many factors influence reflections, including object geometry, material properties, 3D environment, and observer point of view. By deconstructing an object’s geometry and internally brightening it from the specular radiation that reflects off it, humans can derive profound clues and inferences about the shrouded portions of the surrounding environment.

Computer vision researchers at MIT and Rice have developed a way to use reflections to produce images of the real environment. Using reflections, they turn shiny objects into “cameras”, giving the impression that the user is gazing out at the world through the “lenses” of common items such as a ceramic coffee cup or a metal paperweight.

The method the researchers used involves turning bright objects of indefinite geometry into radiation field cameras. The main idea is to use the surface of the object as a digital sensor to record the light reflected from the surrounding environment in two dimensions.

The researchers explain that the new view synthesis, presenting new perspectives that are directly visible only to the bright object in the scene but not to the observer, thanks to the restoration of the environment’s radiation fields. Furthermore, we can imagine agglodrats generated by nearby objects in the scene using radiation field. The method developed by the researchers is taught end-to-end by using several photographs of the object to simultaneously estimate its geometry, diffuse radiation, and radiation field of its 5D environment.

The research aims to separate the object from its reflection so that the object “sees” the world as if it were a camera and records its surroundings. Computer vision has struggled with reflections for some time because they are a distorted 2D representation of a 3D scene of unknown shape.

The researchers model the surface of the object as a virtual sensor, and collect the two-dimensional projection of the 5D environment radiation field around the object to create a three-dimensional representation of the world as seen by the object. Most of the environment’s radiation field is blocked except by the object’s reflections. Beyond the field of view, synthesizing the display of the novel, or presenting new perspectives that are directly visible only to the bright object in the scene but not to the observer, is made possible through the use of the environment’s radiation fields, which also allow for the estimation of depth and luminosity from the object to its surroundings.

In short, the team did the following:

• They show how tacit surfaces can be turned into virtual sensors with the ability to take 3D images of their environments using only virtual cones.
• Together, they calculate the object’s 5D surrounding radiation field and estimate its diffuse radiation.
• They show how to use the light field of the surrounding environment to generate new perspectives invisible to the human eye.

This project aims to reconstruct the five-dimensional radiation field of the ocean from many photographs of a bright element of unknown shape and albedo. Glare from reflective surfaces reveals elements of a scene outside the field of vision. Specifically, the surface rules and curvature of the bright object determine how the observer’s images are mapped onto the real world.

Researchers may need more accurate information about the shape of the reflected object or reality, which contributes to distortion. It is also possible for the color and texture of a shiny object to blend with the reflections. Moreover, it is not easy to discern depth in reflected scenes because reflections are two-dimensional projections of a three-dimensional environment.

The team of researchers overcame these obstacles. They start by photographing the shiny object from different angles, capturing a variety of reflections. Orca (Objects Like Radiance-Field Cameras) is an acronym for their three-stage process.

Orca can record multi-view reflections by imaging the object from different angles, which are then used to estimate the depth between the bright object and other objects in the scene and the shape of the bright object itself. More information about the strength and direction of the light rays coming from and hitting each point in the image is captured by ORCa’s 5D radiation field model. Orca can make more accurate depth estimates thanks to the data in this 5D radiation field. Because the scene is rendered as a 5D radiation field rather than a 2D image, the user can see details that would be obscured by angles or other obstructions. The researchers explain that once ORCa has collected the 5D radiation field, the user can place a virtual camera anywhere in the area and create the synthetic image that the camera will produce. The user may also change the appearance of an item, for example from ceramic to metal, or incorporate virtual objects into the scene.

By extending the definition of the radiation field beyond the traditional line-of-sight radiation field, researchers can open up new avenues for investigating the environment and the objects within it. Using projected virtual widths and depths, the work can open up possibilities in virtual object insertion and 3D perception, such as extrapolating information from outside the camera’s field of view.

scan the paper And Project page. Don’t forget to join 22k+ML Sub RedditAnd discord channelAnd And Email newsletter, where we share the latest AI research news, cool AI projects, and more. If you have any questions regarding the above article or if we’ve missed anything, feel free to email us at [email protected]

🚀 Check out 100’s AI Tools in the AI ​​Tools Club