Leveraging Hololens 2 to empower field service technicians on the job

Context

This is a Master's Program Graduate Capstone Project sponsored by Microsoft Dynamics 365  and supported by Viking Corp.

(Feb - May, 2021)

Methods

Desk Research

Survey + Interviews

Heuristics Evaluation

Bodystorming

Remote Concept Testing

Design Sprint 

Video prototyping

My Role

Researcher

Interaction Designer

Animator 

(On a team of 3 graduate students with Valerie Ijai and Ana M. Villalpando)

Part I. Research Process:

Industry & Product Research

01 Demystify Hololens 2 and Field Service Industry

UX Research

02 Deep Dive into the Target Inspection Workflow

Part II. Design Process:

Design Research & Early Design Brainstorming

03 Research and Brainstorm the Design of TechAssist

Design Ideation

04 Ideate the interactions of TechAssist

Prototyping & Testing

05 Mock up and Iterate the Tech Assist Experience

Augmented workflow guidance

Inspector uses Hololens 2 to scan the marker deep-linked to the fire suppression system inspection workflow

Inspector views the workflow guidance which includes attentional directors, a series of checkpoints, and holographic gems (triggers)

Progressive disclosure of an onboarding flow as inspector walks closer to the first gem

Inspector air-taps the first gem to kick-start the workflow

Instinctive, multi-modal interactions

As inspector reads about the first checkpoint - the bladder tank, she invokes the hand menu using her left hand and taps on “video tutorial” for more guidance using her right hand

Inspector finishes inspecting the first checkpoint and voices “pass” to change its status

3D lifesize overlays

As inspector moves onto the second checkpoint - the valve system, she taps on the 3D Model button on the hand menu to view a series of equipment models in space for reference

Inspector adjusts the size of the h.c. deluge valve model and reads about relevant metrics

With exploded view, inspector can study specific parts of the inner structure of the valve to check the real valve against

Spatial Annotations

Upon noticing anomalies, inspector can choose from a variety of tools to mark up the valve in question

After snapping a shot, inspector can draw on it and use speech to text to verbalize what the problem is about. These annotations will be stored as multimedia files in cloud.

Inspector can view historical annotations of the valve filtered by time and layer them in the original spatial environment in which they were made

Automated synchronization

After indicating that a checkpoint fails the inspection and needs repairing, its status will be updated in both the checklist and automatically synced to the holographic certification tag

The certification tag shows a general overview of the failed checkpoint(s) in the most recent inspection that leads to more details upon request