
Note: *Portions of this entry have been changed to respect a non-disclosure agreement, and is based on publicly available patents (listed at the end). The suggestions and ideas present are my own, and do not represent the views of any companies that I have worked for.*


# Summary

Imagine if your phone was contextually aware of the devices around it, as you are. You could point your phone at the TV, and get contextual applications. You could point it at a smart speaker and have a Spotify widget pop up. Sharing with friends could be as simple as pointing at them, and swiping content in their direction. 

All of this is possible with an emerging technology: Ultra-Wideband (UWB). This radio technology allows for inexpensive local positioning, giving devices such as smartphones and smart watches information about the contextual space around them. 


# Process and Role

For the purposes of this example, we will assume that a platform exists for developing device interactions [link to writing about this].

For this project, the technical and domain areas were _multi-device interaction, ultra-wideband, and spatial computing_. This is an emerging field, with work being done by large smartphone manufacturers such as Apple and Samsung. 

My role in this project was as lead developer and project manager. I was responsible for the entirety of software development as it pertained to platforms and user interfaces. Additionally, I co-developed all interactions for the system with the team director. 


# Research and Design


## Technical Evaluation

This involved testing physical prototypes built by another team member, to determine levels of accuracy and additional areas of improvement such as sensor-fusion optimization. 

The core result was that it is possible to identify devices that the user is pointing a device and devices that are in the regions that the user is pointing a device. 


## Jobs-To-Be-Done

By examining device usage, in the home and workplace, key requirements emerged. 


### Sharing Content

This was a primary task for users, sharing content between themselves and other users, as well as between themselves and a smart device in their home. 

For this, the natural interaction shortcut provided by positioning was device selection. This allows users to skip the often costly step of selecting a target device, making sharing fast and more intuitive. 

::video[User aims a phone at a screen]{#/assets/posts/uwb-smartphone/uwb_interaction_2.webm}

### Application Search

In specific contexts, users wish to use specific applications. Thus, we can limit the search parameters and show users only the most relevant applications. For example, if a user has their device aimed at a Smart TV, we can then suggest TV-related applications such as Disney+ or Netflix.

::video[User aims a phone at a screen]{#/assets/posts/uwb-smartphone/uwb_interaction_1.webm}

### Control

As smart devices become more commonplace, so too does controlling them via a smart device. Selecting that device from either a populated menu or an application is time consuming and slow. It is faster for the user to orient their smartphone at the smart device they wish to control, and give a voice or touch command. 

::video[User aims a phone at a screen]{#/assets/posts/uwb-smartphone/uwb_interaction_3.webm}

## Develop, Iterate, Refine, and Test

Using a platform that I developed and maintained, I was able to quickly mock up user interaction scenarios on a smartphone, with a connected physical prototype. This allowed me to develop, test, and refine interactions for these “Jobs”. A timeline was created, with objectives for the interaction prototypes that were to be developed. 

Building off of the existing android interface, a Card interface was selected to contain the information that a user would want to share. Long-pressing content in the application would prepare it for sharing. Orienting the device towards a target would create a digital space that the user could move the content to with a drag operation. In this digital space, they could manipulate the data that was stored/displayed on that secondary device. 

To give users awareness of the digital landscape and the device they were targeting, a radar interface was developed. Minimized, this would show only when the user was pointing their smartphone at a device. If the user expanded the radar view, they would see an enlarged image showing all of the nearby available devices. In the case where two devices were close together, the radar would show multiple, and the user could select the correct device. 


## Handoff and Prototype

Once the system was developed and tested, an application was compiled, alongside documentation governing its usage and features. Additionally, to best capture it’s capabilities, several use-case videos were created, as well as videos showcasing how to perform specific interactions. 


# Lessons Learned

Spatial computing is a powerful tool that utilizes how the user understands the world, to better facilitate interactions with smart devices. This is an under-explored area in modern smartphone product design, and has much room for growth. 

Second, developing and maintaining a platform to enable rapid prototyping of user interactions was immensely valuable in being able to bring this idea from concept to prototype quickly and effectively. 


# References

Please note that “Sean Braley” translates to 肖恩·布瑞利. 

CN112134995A: Method, terminal and computer readable storage medium for searching application object. 

(https://patents.google.com/patent/CN112134995A/en)

CN112130788A: Content sharing method and device. 

(https://patents.google.com/patent/CN112130788A/en)

CN114095542A: Display control method and electronic equipment.

(https://patents.google.com/patent/CN114095542A/en)


Spatial Smartphone


Note: *This research work took place over the course of several years (and multiple development cycles). It has been modified for brevity and clarity.*


# Summary

The stated intent of the project was to make “magic levitation”. The exact form this would take was left open for interpretation. This system is an _interactive_ 3D shape changing display. At its core, a _programmable matter_ system. 

This is an emerging field. The greater majority of work on shape changing interfaces has, and continues to, focus on _supported_ display mechanisms. That is, the elements of the system have some form of secondary support enabling their position. The most common form of these are so-called 2.5D systems, where part of one “D”, often the table, is constrained as a platform for mechanization. These systems are useful, but in order to develop and understand interactions in a _fully 3D system_, each element must be self-levitating.

Self-levitating systems are capable of unrestricted movement in all three dimensions, either independently or through input from the user. The goal of this project is to support both. Additionally, individual voxels (3D pixels) were required to be capable of displaying an RGB color. 


# Process and Role

First principles design research in the interaction space is founded upon the process of determining an area of research (shape changing displays), then developing a novel technological platform for that area (indoor safe-to-touch drone swarm), then developing interactions for that system (touch-to-drag). 

Once the area of research and technological platform are determined, the sub-process of technological and interaction design happens. Especially with novel systems, these operate hand-in-hand as capabilities determine interactions, and interactions determine system design parameters. 

My role in this project evolved over time, from a contributor, to lead interaction designer, system architect and primary developer. For this project I developed skills in working with drones, local positioning systems, control systems, and real time systems. I was able to utilize my previous experience with software architecture and interaction design. 

The drone units were built by a colleague Calvin Rubens, and then broken by myself. The research was under the direction of Roel Vertegaal (http://humanmedialab.com). 

Additional researchers on this project include: 

Antonio Gomes

Xujing Zhang


# Research and Design


## Technical Development

The system utilizes quad-rotor micro-drones. Each is housed within a wireframe cage allowing safe touching and manipulation. Additionally, each is equipped with a lightweight LiPo battery for power, color changing LED for display, and WiFi radio for communication and control. 

![alt_text](/assets/posts/bitdrones-os/bitdrone.jpg "micro drone with labels showing the parts")

### Technical Challenge: Control and Positioning

Small drones, operating indoors, cannot utilize GPS systems, as they are neither accurate nor light enough. The most-viable method of control for &lt;50g drones was a Local Positioning System (LPS), which would connect to a system running individual PID-based controllers for each axis of the drone’s position. 

The system utilized a Vicon system, with retroreflective markers being placed on each frame. This would feed the position at a rate of 100Hz to the control system, which would in turn send control instructions to the individual drone systems over a wireless network. 


### Technical Challenge: Small Drones

In order for the units to form individual pieces of matter in a programmable matter display, they were designed to be as small as possible, while still maintaining a flight time of at least five minutes. Motors, propellers, and batteries all become less efficient per gram at smaller sizes. This resulted in a tradeoff sized unit where the individual drone could be grabbed by the average hand. 


### Technical Challenge: Testing With Real Systems

Interaction design must, fundamentally, be tested “in the real”. This meant that the drone system was operational, in some capacity, for the majority of development. Of course, there were instances where the system would fail, and the drones would lose control. This was a serious challenge, and was overcome by hardening the flight control and response systems first. 


::video[Drones fall from the sky]{#/assets/posts/bitdrones-os/spin_cycle.webm}

## Interaction Design Development


### Standard Interactions Translate

Intuitive gestures with the drone system such as tap-to-select, drag-to-reposition, and two-hand-drag all translate naturally to the 3D interaction space. 

::video[User touches a drone]{#/assets/posts/bitdrones-os/drones_touch.webm}


### Natural Discovery: Built-In Elasticity / Force Feedback / Physics

Pushing on a drone that is doing its best to stay in a given position gives a sense of force-feedback that is intuitive to users. Following, then, that if we _allow_ a user to move a drone, its resistance is limited as its “new desired position” is updated continuously as the user moves it.


### Natural Discovery: Play

Unencumbered by the fear of larger, more powerful drones, and with their components safely shielded away, participants of all ages found great pleasure in simply “playing” with the drones. This was an obvious phenomenon witnessed in all demonstration scenarios. Users would enjoy moving, dropping, and tapping the drones. 


### Secondary Interactions

Certain interactions (such as color picking), a higher resolution display was required. For this, a smartphone was used with a simple android application as a stand-in for a user’s cell phone. For telepresence, or remote collaboration, a larger drone which had a flexible display was used, along with a video calling application. 

An additional interaction mechanism was developed for _remote_ interactions with the drones. A wand that could be tracked by the local positioning system. Interactions with this device included ray-select, lasso-select, point-and-click, click-and-drag (remote).

![alt_text](/assets/posts/bitdrones-os/skype.jpg "user speaks to a drone in a video conference")

## Documentation and Demonstration


### Research Papers, Video and In Person Demonstration Events

The research papers for this system contained detailed information on the system construction and the interactions presented. The interactions were best presented in a video or picture format, where the configuration of the system could be seen. 

Logistically, preparing and moving a drone system for demonstration is no small task. The system is, technically, “portable”. For it to be moved, a local positioning system must be either moved with it, or made ready at the other end. The drones themselves are relatively fragile and must be carefully packed. LiPo batteries are dangerous and must be transported carefully. 

In the demonstration location, power and space are required. Additionally, as most LPS systems rely on infrared, there cannot be too much ambient sunlight. Finally, WiFi congestion can cause issues with the drone communications. 


# Lessons Learned

A uniformly top-down control system was quicker to develop and debug. But, a system with distributed control would have been more robust. It would also be able to take advantage of dead-reckoning to handle intermittent positioning information loss. 

A system to dock and charge the drones would have decreased the time spent swapping and managing batteries. Combined with a landing routine, this would have made the system completely automated. 


BitDrones


Note: *Some details have been changed and specifics omitted to respect a non-disclosure agreement.*

# Summary

As is disturbingly common in many growing businesses, the backbone of the product management department was an Excel spreadsheet. Until a certain point, the flexibility offered by this was a worthwhile compromise, but it had several drawbacks. 

First, versioning and version control. Hosted locally or on a network drive, as this was, version and access control presented an issue. Second, interfacing with other tools. This spreadsheet was used to maintain a record of the key product information such as MSRP and COGS, and used for business evaluations such as gross margin. While these can, and often are done by hand, interfacing with sales tools again relied on this data being manually imported.

I was recruited to modernize this into a flexible tool, which could fill several other holes in the product management process within the company. 


# Process and Role

The design process followed a jobs-to-be-done format, where I would first investigate the utility provided by the excel spreadsheet, then determine how to translate and extend those functionalities in a custom application. 

There were several key stakeholders in this development. Primary was the product department for which the tool was developed. They defined the key business processes that would need to be replicated. Within this department, requirements emerged from technical staff on the hosting and data management, and from the product department on exact definitions of tools. Unconstrained by the limits of a spreadsheet, additional requirements emerged from the product department. Secondary was an outside developer, who created and maintained a custom sales tool.

I was the project manager and primary developer for this project. I performed the use-case analysis, as well as data structure design. Additionally, I was responsible for the testing, deployment and management of the tool. I interviewed users of the Excel spreadsheet, and determined what they were “hiring” it to do. 


# Research and Design

The requirements for this tool naturally led to an authenticated web application. This would allow multiple users to edit a shared database, while maintaining a coherent, traceable and auditable base. 


### Compare Existing Solutions

There were two primary factors for using an in-house developed system. First was company culture, which preferred such solutions. Second, was the challenge in finding something that delivered _all_ of the required features. 


### Base Features

Pages were required for searching products, and editing them. These pages should load quickly, perform the expected task, and allow for intuitive control. It was requested to add features beyond the capabilities of the spreadsheet, including items such as a “depends on” relationship between items. 


### Core Feature: PDF Datasheet

Previously, all datasheets were manually generated by the marketing department. This would require regeneration when (any) feature or property of a product changed. This would lead to product datasheets being out of date, or using the wrong format, etc. Because the data was structured, it was then easy to format this into a PDF export functionality, allowing always-up-to-date datasheets. 


### Core Feature: Excel Export

Excel is, in many ways, inescapable. For some business processes, be those official or “how someone does something”, there are hard requirements that clash with the existing or proposed solution. Interaction designers have two solutions at this point: Ignore the requirement and hope it goes away, or incorporate it into your design. 

To take an illustrative example, at first it seems at odds with the initial goal of _getting away from spreadsheets_ to offer an export-to-excel function. However, the _real goal_ was to create a unified, controlled and auditable system for product management. Having an export tool simply allows a _copy_ of that data to be exported for convenience. At no point does the _exported copy_ become the primary. In this case, the solution was not a code-solution, per-say, but a business process one.


### Core Feature: Audit Tracking

Every change made to product information was to be logged and associated with an account. This was a feature that set the tool apart from simply placing an excel spreadsheet in a cloud-provider platform. Accountability and traceability were important for a company seeking to maintain ISO certification. 


### Platform and Tools

This system was developed as a Python web-application, running on a Linux server, connected to a Microsoft SQL database. 

This project utilized the TurboGears web framework, which was chosen at the time for being quickest-to-build and was a competing product to Django. SQLAlchemy was used as the backing ORM, as it gave a good level of abstraction between the SQL backend database and the higher-level Python code. 


![alt_text](/assets/posts/project-management-editor/structure.png "platform structure")

_Platform Structure_


# Lessons Learned

The number one takeaway from this project was the directness with which jobs-to-be-done theory could be applied to business processes. Users had very specific routines that they were “hiring” the spreadsheet for - and many besides that they _wanted to_. 

The second lesson has been the longevity of the tool. In research, things are often built as one-offs, or outdated by new developments. This tool has been in successful operation for nearly a decade.

Finally, seeing the impact on business processes over a long time-span has been a key driver in valuing the importance of creating systems for _the humans that use them_, and not just for the business process. 

Research Process

Featured Work

Spatial Smartphone

BitDrones

Project Management Editor