Wearable cognitive assistance applications  are a new class of applications that combine computer vision and edge computing and are designed to aid in the completion of complex or involved tasks. At Carnegie Mellon University, we have developed the Gabriel platform for building wearable cognitive assistance applications. Below are some of the various applications we have created including links to the source code, docker containers encapsulating the backends, and demonstration videos of the assistants.

Gabriel Platform

The Gabriel repository contains the core backend code as well as implementations for several clients, including an Android client (available on the PlayStore), a Python client, and Microsoft Hololens client.

Source Code

Docker Containers

Lego

Lego was the very first wearable cognitive assistant ever developed. It uses a special backboard to detect the color/shape of various Lego pieces and helps the user to construct a specific Lego kit.

Source Code

Demo Video

Sandwich

The idea behind sandwich was to develop a wearable cognitive assistant to help with cooking. The application gives instructions to the user on the order in which they should assemble the components of the sandwich. For this demo, a Microsoft Hololens client was also created which shows a holographic representation where the next item needs to be placed.

Source Code

Demo Video

Ikea

A universally understood challenge is building furniture. To address this challenge, a wearable cognitive assistant was designed for an Ikea lamp. Through short video clips and audible instructions, users can quickly put together the lamp.

Source Code

Demo Video

DiskTray

In cooperation with inWin, we developed an assistant to aid in the assembly of a hard disk tray prior to incorporating it into the computer chassis. Instead of reading written instructions, the user wears a pair of smart glasses (ODG-R7) and is given step by step video and audio guidance.

Source Code

Demo Video

RibLoc

RibLoc is a Gabriel application designed to assist  for surgical repair of ribs. The RibLoc system  is made by AcuteInnovations, Inc. Today, this training is given to a doctor by an on-site technician. This wearable cognitive assistant illustrates how this training could be delivered more efficiently.

Source Code

Demo Video