Touchless computing could improve computer use for millions

Software allowing users to control their PC with a webcam, using intricate movements in their face and body, has been developed by academics and students in UCL Computer Science and could revolutionise the way millions of people use computers.

The software, UCL MotionInput Version 3, replaces the input functionality provided by a keyboard, mouse or joypad, allowing users to control and interact with a computer using movements of their hands, fingers, head, eyes, nose, eyebrows, mouth or full body, captured by their webcam. They can also simultaneously give spoken commands and enable live transcription, for example to fill in a website form, captured by their computer microphone.

The software was first designed by UCL academics and students in response to the Covid-19 pandemic to help prevent the spread of viruses in clinical settings, by enabling medical staff to use communal computers without touching the keyboard or mouse.

It has since been reviewed by further NHS doctors and clinical teams at Great Ormond Street Children’s Hospital (GOSH) and advocated by charities including the International Alliance of ALS/MND Associations, which supports people with Motor Neurone Disease (MND) around the world.

The UCL team have been developing the software for over two years in collaboration with industry leaders including Intel, Microsoft, IBM, Google and the NHS via the UCL Industry Exchange Network in UCL Computer Science, as well as GOSH and UCLH Institute for Child Health.

Project lead Professor Dean Mohamedally (UCL Computer Science), who supervised the student teams along with Professor Graham Roberts and other UCL academics said: “UCL MotionInput is the most advanced touchless computing software for use currently available with just a regular webcam and a Windows PC. It utilises and combines the latest open-source machine learning, natural language processing and computer vision software from globally recognised tech firms like Intel and Google, in a way that hasn’t been done before. Previously, people with mobility issues had to buy adapted computers and use pointer devices in order to type or move the mouse cursor.

“This is a unique software that can be used on the majority of Windows PCs and doesn’t require any specific equipment other than the webcam, which most laptops already have, meaning there are very few barriers to access. The software is still maturing, but with a supportive community collaborating with us, we hope to see it flourish with more user needs met. We hope this can make computing more accessible and enjoyable to millions, as well as support clinical staff in environments where they are at a higher risk of catching and transmitting diseases and viruses.”

Once downloaded, the software can turn any screen into a multi-touch, in-air touchscreen, meaning it can interact with a variety of different movements from the user. As well as its clinical uses and for those with mobility issues, the software can also be used for teaching and presenting in schools, universities and businesses, and by people playing Windows computer games.

Users can also give pre-programmed commands, for example, saying “click”, “right click” and “double click” to command those actions.

This year’s version of the software was developed by 54 UCL students across multiple year groups and courses at UCL Computer Science. Final year student Sinead V Tattan led development, orchestrating the students to build the software components and bring it together. Carmen Meinson, a second-year student, worked to highly optimise the software architecture.

The team also worked with PhD candidate Sheena Visram (UCL Interaction Centre, UCLIC) and Dr Atia Rafiq, an NHS GP and UCL Computer Science honorary lecturer, to ensure the designs met clinical requirements.

Ms Tattan said: “For version 3.0, Professor Mohamedally tasked us with creating something that would really make a difference in people’s lives – an application that could give those with fine motor skill conditions opportunities that typically abled people take for granted. We divided into teams to take on specific functionalities, such as developing the world’s first hybrid facial navigation with simultaneous voice commands. We then had to ensure they could be combined and accessed as individual software features or a single easy-to-use suite of programs.”

Bernice You (General Manager, Strategy & Projects, Microsoft) said: “With technology we can empower everyone and make the impossible possible. Love the potential and possibilities that MotionInput brings to people with disabilities.”

Cathy Cummings (Executive Director, The International Alliance of ALS/MND Associations) said: “MotionInput improves quality of life and maintains independence and dignity by making things that matter accessible.”

The software is freely available to download for individual users, and the team hope to see it widely rolled out across NHS sites as well as schools and other industries looking to invest in the technology. The team are also keen to connect with user groups with various accessibility and education requirements in supporting equitable computing access for all.

MotionInput can be downloaded here

About UCL – London’s Global University

UCL is a diverse global community of world-class academics, students, industry links, external partners, and alumni. Our powerful collective of individuals and institutions work together to explore new possibilities.

Since 1826, we have championed independent thought by attracting and nurturing the world’s best minds. Our community of more than 43,800 students from 150 countries and over 14,300 staff pursues academic excellence, breaks boundaries and makes a positive impact on real world problems.

We are consistently ranked among the top 10 universities in the world and are one of only a handful of institutions rated as having the strongest academic reputation and the broadest research impact.

We have a progressive and integrated approach to our teaching and research – championing innovation, creativity and cross-disciplinary working. We teach our students how to think, not what to think, and see them as partners, collaborators and contributors.  

For almost 200 years, we are proud to have opened higher education to students from a wide range of backgrounds and to change the way we create and share knowledge.

We were the first in England to welcome women to university education and that courageous attitude and disruptive spirit is still alive today. We are UCL. | Follow @uclnews on Twitter | Read news at | Listen to UCL podcasts on SoundCloud | Find out what’s on at UCL Minds