Toward better web based phone VR

[Thomas Sharpless]

The conventions of VR have so far been shaped mainly by gaming and simulation. But other applications of binocular digital display technology are equally important if not more so. Examples: immersive 3D movies, virtual tourism, popular 3D photography. Those applications can work nicely on a phone, and don’t need the kind of fast interface devices that gaming and simulation do. Phone VR can be delivered equally well by web technology as by proprietary apps. Although apps are dominant now, I believe the web route will eventually become the more important market.

Right now, web-based phone VR is in a serious early-adopter crisis, basically due to the major inconveniences of using it. With HTML5 and WebGL, javascript players like krpano now do a good job of showing binocular stills and video, and provide some rudimentary tools for navigating within a packaged tour. But a satisfactory web VR experience demands a lot more. We need better tools, and rules, for interacting with phone VR displays, and for browsing the web while the phone is in a VR viewer.

A big part of the problem is lack of practical user input devices that work when the phone is “caged”. Only the phone’s gyros are really useful; its major UI device, the touch screen, is partly or completely blocked. Some VR viewers do let you put a finger or two on the screen, but with limited access, and the screen just inches from your eyes, and a split view, you really can’t do much.

We need some radical thinking about how a caged phone could/should relate to web content and the web itself. Can we really do without a 2D tracking device, when so many established UI conventions depend on one? Can we really eliminate our browsers’ need for text input? Can we devise a subset of HTML5 that supports basic web navigation without requiring either tracking or text? I doubt it, though I’m willing to be pleasantly surprised. But it may be more practical to think of alternate ways to provide the lost touch screen functions.

One solution is to equip the cage with UI devices, as Samsung has done on the GearVR. That is a viable approach, but at present a proprietary one, that works only on a few phone models. And the current GearVR UI is not really up to the job of running a web browser because it has too few buttons and no easy way to input text.

It should also be possible to exploit devices already built into the phone, or supported as peripherals by its operating software. All phones can hear, so voice control is one possibility. They can also see, so part of the VR UI might involve waving your hands in front of the phone. Or, since the front-facing camera is likely to be preempted for AR purposes, it might be possible to use the selfie camera as a control device. All Android phones have some support for Bluetooth keyboards, game pads and tracking devices; and if the demand was there, iPhones could, too.

All of the above possibilities need new or improved software support in the OS, perhaps through add-on drivers, before a web browser could use them. It is likely that Bluetooth peripherals would be easiest to adopt since they already have well defined APIs in most OSs.

Perhaps the best solution would be for phone makers to put a trackpad and some tap-sensitive buttons on the back side of the phone, just for use in VR, and integrate those in their OS ‘mouse’ and ‘keyboard’ APIs. Or maybe a 3rd party could make a “stick-on VR pad” and the requisite drivers.

[Thomas Sharpless]

Update

The just announced Google Daydream controller is a big step forward for phone VR. It is a hand held device, coupled to the VR phone by Bluetooth, that has a clickable trackpad and two buttons. In addition it has gravity and magnetic sensors and rate gyros, so the VR phone can read its orientation and speed. This opens a lot of possibilities for interaction. Vive-like gaming, of course. But hopefully this device will also stimulate the development of a standardized gesture-based browser UI. Bravo, Google!

[Author]

Posts