360-degree videos and images in Unity

  1. Import the 360-degree video into Unity as an Asset.
  2. Create a Video Player by dragging the video asset to the scene hierarchy
  3. Render the video to a Render Texture for precise display control: create a Render Texture from the Assets->Create.
  4. Set the Render Texture’s Size to match that of the video exactly (the dimensions of the video are given under the “Source Info” section in the Preview UI in the Video Asset Inspector)
  5. Set the Depth Buffer option to No Depth Buffer(no need for depth information).
  6. Switch the Render Mode to Render Texture and drag the newly created Render Texture from the Asset view to the Target Texture slot. 
  7. Create a Skybox Material: In order to render the panoramic video as a backdrop to the scene, replace the default Skybox with the video content. Create a new Material from Assets->Create.
  8. Set the Material’s Shader to Skybox/Panoramic.
  9. Drag the Render Texture from the Asset view to the texture slot in the new material.
  10. In order for the panoramic video to be properly displayed, the type of content in the video must be set correctly For cubemap content (such as a cross and strip layout as is common for static skybox textures) select the 6 Frames Layout Mapping. For equirectangular, choose Latitude Longitude Layout Mapping and then either the 360 or 180 degree sub-option depending on if the video covers a full 360 degree view, or just a front-facing 180 degree view.
  11. Pan around in the Preview at the bottom of the Material inspector and check that the 360 or 180 content looks correct.
  12. Connect the Skybox material by opening up the Settings panel from the Window->Lighting menu. Simply drag and drop the new Skybox material asset to the first slot under Environment. Changing the scene camera orientation will show a different portion of the Skybox and therefore a different portion of the panoramic video.
  13. If you have Virtual Reality Support turned on in the Player Settings, an extra 3D Layout option will be available in the Skybox/PanoramicBeta material. If your source video has stereo content this option should be turned on. Use the Side by Side settings if the video contains the left eye’s content on the left and the right eye’s content on the right, or, choose Over Under if the left and right content are positioned above and below one another in the video. The correct half of the video will be used when rendering each eye’s content in VR.

Differences between equirectangular and Cubemap 360 videos

Cubemap (6 Frame Layout) 2D videos should have aspect ratio 1:6, 3:4, 4:3, or 6:1 depending on face layout. Equirectangular (Latitude Longitude Layout) 2D videos should have an aspect ratio of exactly 2:1 for 360 content or 1:1 for 180 content. Including 3D content will require double either the width or the height of the video (corresponding to Side-by-Side or Over-Under layouts). Keep in mind that many desktop hardware video decoders are limited to 4K resolutions and mobile hardware video decoders are often limited to 2K or less which limits the resolution that can be played back in realtime on those platforms. Video transcoding can be used to produce lower resolution versions of panoramic videos but precautions should be taken to avoid introducing bleeding at the edge between left and right 3D content, or, between cube map faces and adjacent black areas. In general, authoring video in power-of-two dimensions and transcoding to other power-of-two dimensions is suggested to reduce visual artifacts.

Equirectangular

Equirectangular is the traditional layout used for 360 videos. The problem is that this layout can contain redundant information at either end. Think of it in terms of a map of the globe. Antarctica is really a circular landmass, not a drawn-out linear one. How you display the map affects how much extra Antarctica there is in the image.

In 360 video, instead of a warped landmass, there are either same-colour pixels nearby or pixels that are indistinguishable at the render time. A solution to this problem is to remap equirectangular projection layouts to cube maps.

Cube map

Cube map projection is a combination of six faces of the cube. Cube maps have been used in computer graphics for a long time, mostly to create skyboxes (six-sided cubes that are drawn behind all graphics) and reflections. There are a few benefits of using cube maps for videos:

  • Cube maps don’t have geometry distortion within the faces. So each face looks exactly as if you were looking at the object head-on with a perspective camera, which warps or transforms an object and its surrounding area. This is important because video codecs assume motion vectors as straight lines. And that’s why it encodes better than with bended motions in equirectangular.
  • Cube maps’ pixels are well-distributed — each face is equally important. There are no poles as in equirectangular projection, which contains redundant information.
  • Cube maps are easier to project. Each face is mapped only on the corresponding face of the cube.
Advertisements
%d bloggers like this: