I’ve worked in the technology space for 25 years and also taught technologies for tangible interaction design as a part-time sessional lecturer at Monash University for the last 5 years, and as a result, helped hundreds of students create prototypes of their tangible interaction systems.

What are Tangible Systems?

Tangible systems combine software systems with physical inputs or outputs to enhance human experience or capabilities in novel ways.

• When the user’s input and feedback loop is tangible, it is called a tangible interaction
• when the output is tangible, say in a printer, that’s a tangible action.

Simple examples of tangible systems are touch screen devices like GPS navigation, motion-based controls (think Nintendo Wii). More elaborate product examples are HoloLens AR Glasses, Virtual Reality, IoT based assistants like Amazon Alexa, Amazon’s warehouse robotics, etc. Massively successful tangible systems are iPhone, iPad, Tesla, Nintendo Wii, Oculus Rift and HTC Vive.

What is NOT a Tangible System?

Computing is generally an imaginary thing. You type a document and it does not exist in the real world until it is printed. The interactions you have with a Bold button in Microsoft Word is you clicking on the mouse button when the mouse cursor is hovering over the B Icon. It’s just not real - it is all an imaginary, conceptual interaction.

By comparison, virtual reality headsets use the motion of the user’s head to control the motion of a virtual camera. The experience is more tangible for the end user because the link between their action and the visible reaction is one-to-one.

Examples of Applied Tangible Systems

Mobile Phones Reinvented - iPhone

Prior to the launch of the original iPhone, the leading handsets were made by Nokia and Blackberry. They were phones primarily, simple messaging devices and maybe you could check your email. There were some Personal Digital Assistant phones based on Windows CE, commonly referred to as Wince for the poor user experience, but they didn’t change the market much.

June 2007 Steve Jobs launched the original iPhone, Apple’s first cellular phone and disrupted the category. It was the first phone to have a high-resolution touch screen that supported multiple touches for gestures. In late 2007, there were about 5,000 iPhones using the Telstra network before they were officially supported by Telstra.

By removing the physical keyboard that dominated a handset and replacing it with a touchscreen, the iPhone provided extra utility value because it could also be used to consume internet media, play music and surf the web. More importantly, the number pad or keyboard on the iPhone worked like a real one because of the touch screen but it could also disappear and let the screen just be a screen.

Digital Cameras

Digital photography technology had been in development since the mid 1970’s but it wasn’t until the end of the 20th Century that they were launched as consumer products. However, once they were launched as consumer devices, digital cameras defined photography. They were simple to use and made photography much more accessible and affordable than ever before by removing the cost of film and developing chemicals. When digital cameras were available for the mass market, they completely disrupted the photography industry.

Video game engines are becoming tangible systems in the movie production process by supporting VFX, pre-visualisation and near real-time feedback.

Pokémon Go - An Augmented Reality Experience

The July 2016 Business Insider article 14 things you didn’t know about Pokémon Go and how it was made shows just how big Pokémon Go was and how it evolved as a business enterprise. The Verge reported that Pokémon Go added $7.5b US Dollars to Nintendo’s market value, growing the shareprice by 24.52% in a single day.

Tesla Redefines the Automotive Segment

A Tesla vehicle is not just an electric car, it is a sophisticated tangible system comprised of tangible interactions and tangible actions. The car has control systems that could rival that on an aeroplane, with autopilot, interactive maps and crash avoidance. A Tesla car has multiple tangible interactions, not only through its integrated touch screen that replaces internal buttons and knobs but also through a mobile phone application. The crash avoidance, autopilot, the heating systems and door locking mechanisms are all tangible actions.

Deep Learning and Costs

Machine learning and deep learning are heavily in use for data processing and business intelligence but they have much more value by bringing tangible systems closer to a human way of understanding the world.

In the early 1990’s Yann Le Cun and other researchers devised an artificial neural network architecture that could learn to recognise handwritten digits. This application of Machine Learning is touted to have inspired a resurgence into the field, leading to deep learning.

This 2018 blog entry on deepsense.ai discusses examples of business savings due to the training and operation of deep learning control systems implementing tangible actions.

Robots in the Warehouse

This blog entry by Logiwa discusses manufacturing and warehouse robots and how they lead to labour cost savings.

This Wired article discusses the operational improvement of Amazon’s warehouses and sorting facilities thanks to its introduction of robotics. For Amazon, these robots are the key to Amazon’s same-day delivery strategy. For Amazon, it’s not about saving money but enabling more efficient and effective delivery.

Where are the Opportunities?

There is a new intersection of cloud services, mobile computing and sensor technologies that create new opportunities for remote inspection systems, machine learning for autonomous process control and continuous improvement, spatially-based interactions for mental health, retail or entertainment contexts. The costs are falling, the tools are improving and the technologies are more accessible than ever before.

Chances are that almost any product or service offered on the market could be reinvented using tangible systems.

Optimise Where There is Volume

Computerising or automating a process can be beneficial both because it can create savings but also because any process improvements can be written in code and implemented quickly and consistently. As a result, scaling up can be achieved and allow a company to compete on volume.

Automation for Building Capability

Once processes are automated, the software development practices should include the latest Continuous Integration / Continuous Delivery automation so process improvements can be incorporated into the software, automatically tested to avoid regressions and rapidly deployed.

Machine learning can be used for quality assurance inspection at multiple points in a production process. Error rates can be correlated to process changes made in automation, changes in the supply chain or linked back to machine maintenance timing.

Automation of this nature is part of an innovation culture.

Shake Up Old Assumptions

Often there are assumed limits to the way you operate. For instance, in technology you don’t deploy a change on a weekend for fear that it will fail and no one is available to correct it. However, if being able to release software more often made your business more competitive, mitigation strategies can be implemented. For example, monitoring systems can be used that would rollback a change if it were to fail. Blue/green deployment approaches can be used… it’s really a cost/benefit consideration.

Automation and software practices can be used to do the things your industry won’t and use that for differentiation.

Ease of Use - Delight the User

Tangible interactions can make a complex, user-guided process more intuitive by making the user input more direct or by improving the user’s view.

Direct inputs in tangible interactions, like swiping on a touch screen to scroll text, doesn’t require a manual to learn because it is the kind of control that is naturally discovered and not easily forgotten.

Summary

Tangible interactions have been used again and again to:

• vastly improve the user experiences of products
• redefine the product category and disrupt an industry
• reduce operational costs
• reduce training costs
• empower people to do more meaningful work

The technologies needed to build tangible systems are more affordable and available than ever before. Devices like iPad and iPhone have made touch and high resolution screens of various sizes more available. This is what made the current generation of VR headsets possible.

Cloud services offer simple mechanisms to build and train Deep Learning models for image recognition for use in software applications. APIs exist to use pre-built machine learning services for voice recongition, intent recognition for chat bots, image recognition and more.

The trick is to evolve the automation where the risks of failure come with high stakes:

1. create a reasonable process monitoring system
2. make it smarter with machine learning and automated sensors where possible
3. partially automate within the limitations of the monitoring and have it trigger automated error handling
4. increase the automation toward full process

Photo Credits

• VR Guy in Black T-Shirt - by Minh Pham
• Lady operating iPad with Laptop in background - by Marek Levak