Back to Portfolio
Spatial Smartphone
Note: Portions of this entry have been changed to respect a non-disclosure agreement, and is based on publicly available patents (listed at the end). The suggestions and ideas present are my own, and do not represent the views of any companies that I have worked for.
Summary
Imagine if your phone was contextually aware of the devices around it, as you are. You could point your phone at the TV, and get contextual applications. You could point it at a smart speaker and have a Spotify widget pop up. Sharing with friends could be as simple as pointing at them, and swiping content in their direction.
All of this is possible with an emerging technology: Ultra-Wideband (UWB). This radio technology allows for inexpensive local positioning, giving devices such as smartphones and smart watches information about the contextual space around them.
Process and Role
For the purposes of this example, we will assume that a platform exists for developing device interactions [link to writing about this].
For this project, the technical and domain areas were multi-device interaction, ultra-wideband, and spatial computing. This is an emerging field, with work being done by large smartphone manufacturers such as Apple and Samsung.
My role in this project was as lead developer and project manager. I was responsible for the entirety of software development as it pertained to platforms and user interfaces. Additionally, I co-developed all interactions for the system with the team director.
Research and Design
Technical Evaluation
This involved testing physical prototypes built by another team member, to determine levels of accuracy and additional areas of improvement such as sensor-fusion optimization.
The core result was that it is possible to identify devices that the user is pointing a device and devices that are in the regions that the user is pointing a device.
Jobs-To-Be-Done
By examining device usage, in the home and workplace, key requirements emerged.
Sharing Content
This was a primary task for users, sharing content between themselves and other users, as well as between themselves and a smart device in their home.
For this, the natural interaction shortcut provided by positioning was device selection. This allows users to skip the often costly step of selecting a target device, making sharing fast and more intuitive.
Application Search
In specific contexts, users wish to use specific applications. Thus, we can limit the search parameters and show users only the most relevant applications. For example, if a user has their device aimed at a Smart TV, we can then suggest TV-related applications such as Disney+ or Netflix.
Control
As smart devices become more commonplace, so too does controlling them via a smart device. Selecting that device from either a populated menu or an application is time consuming and slow. It is faster for the user to orient their smartphone at the smart device they wish to control, and give a voice or touch command.
Develop, Iterate, Refine, and Test
Using a platform that I developed and maintained, I was able to quickly mock up user interaction scenarios on a smartphone, with a connected physical prototype. This allowed me to develop, test, and refine interactions for these “Jobs”. A timeline was created, with objectives for the interaction prototypes that were to be developed.
Building off of the existing android interface, a Card interface was selected to contain the information that a user would want to share. Long-pressing content in the application would prepare it for sharing. Orienting the device towards a target would create a digital space that the user could move the content to with a drag operation. In this digital space, they could manipulate the data that was stored/displayed on that secondary device.
To give users awareness of the digital landscape and the device they were targeting, a radar interface was developed. Minimized, this would show only when the user was pointing their smartphone at a device. If the user expanded the radar view, they would see an enlarged image showing all of the nearby available devices. In the case where two devices were close together, the radar would show multiple, and the user could select the correct device.
Handoff and Prototype
Once the system was developed and tested, an application was compiled, alongside documentation governing its usage and features. Additionally, to best capture it’s capabilities, several use-case videos were created, as well as videos showcasing how to perform specific interactions.
Lessons Learned
Spatial computing is a powerful tool that utilizes how the user understands the world, to better facilitate interactions with smart devices. This is an under-explored area in modern smartphone product design, and has much room for growth.
Second, developing and maintaining a platform to enable rapid prototyping of user interactions was immensely valuable in being able to bring this idea from concept to prototype quickly and effectively.
References
Please note that “Sean Braley” translates to 肖恩·布瑞利.
CN112134995A: Method, terminal and computer readable storage medium for searching application object.
(https://patents.google.com/patent/CN112134995A/en)
CN112130788A: Content sharing method and device.
(https://patents.google.com/patent/CN112130788A/en)
CN114095542A: Display control method and electronic equipment.