Augmenting Assembly Line Operations Through Multimodal Interaction

Umeå Institute of DesignSweden

Disrupting, Empowering

Concept / Student


Siddharth Hirwani


Over the past century, we have made great progress in the development of assembly lines. The nature of assembly line work has always known to be repetitive and monotonous and one that is not just physically demanding but also mentally challenging. Although most of the operations on the line have been automated, the final assembly process, especially in large industries, still requires the agility and speed of humans working on them. Training operators for these complex tasks is both expensive and time intensive for companies to invest in. This makes cross training operators for more than one sub-assembly setting difficult. Lack of cross training is one of the main causes of monotony. Monotony also leads to mental exhaustion and that is when most mistakes happen.

The resulting concept is an exploration into how multimodal interaction, including augmented reality, haptic and conversational interfaces, could seamlessly meet the challenges of training and accomplishing complex tasks. Humans are by nature multimodal, hence an interface to assist users by managing their cognitive load and minimizing performance errors supports a humanized training system.

Although there are existing solutions trying to use augmented reality (AR) as a tool in the manufacturing industry, it is only focused on the visual elements which immediately leads to visual overload. Distributed information and interfaces allow users to intuitively understand and easily use such technology. To interact with existing AR concepts, one needs to make gestures in the air and turn one’s head and look at objects which are quite awkward and inconvenient but also potentially dangerous in an assembly environment. My proposal includes a haptic glove that has touch surfaces that act as a local point of interaction and navigation for the entire system but also to receive tactile feedback while performing tasks.

The multimodal solution allows workers to be able to jump onto tasks immediately and the system provides step-by-step assistance to the operator. As the operator begins to learn, the level of assistance drops into the background only to come back up when the operators make a mistake. Information that might initially be visual, can overtime be relayed via audio or even haptics based on the complexity or the action.

Once an operator has been working on a sub assembly line for a while without making significant errors, they can then switch to another sub-assembly with the system again beginning a new training session. The resulting super operators are now trained across various tasks and are no longer limited to one repetitive activity. They can gain skills and experiences helping them fulfill their potential, ultimately increasing satisfaction, thus humanizing the assembly work.

Using multimodality for training operators does not just create a new way to teach people but also allows for a natural way of interacting with emerging technologies such as augmented reality.



Interaction Awards is an initiative of the Interaction Design Association (IxDA), a global community of over 100,000 individuals worldwide dedicated to the professional practice of Interaction Design. Find out how to join your local group and get involved at

© 2012 - 2018 Interaction Design Association