Off-highway machinery operates in demanding environments with low visibility and tight spaces, requiring operators to always remain focused and alert. As automation advances, the role of the operator and operator assistance systems is evolving. The EU-funded THEIAXR research project explored the use of extended reality (XR) technologies, which include virtual reality (VR), augmented reality (AR), and mixed reality (MR), for industrial applications to enhance operator comfort and safety. TTControl led the project consortium and worked with long-time customer Prinoth in a pilot use case with snow groomers.
The THEIAXR project started in January 2023 and ran for three years, with the objective of making the invisible visible and improving human-machine interaction in mobile machinery with XR technologies. The project brought together ten consortium partners from six European countries – from off-highway machinery manufacturers and suppliers to universities and research institutions. Their goal was to develop an XR interaction concept and principles for the intuitive design of XR interaction in industrial applications. XR is used to provide operators with active feedback from their surroundings, making them aware of potential risks, while improving efficiency and comfort. The resulting design was then validated across three use cases – snow grooming, logistics, and construction.
“THEIAXR stands out thanks to its human-centric approach and transdisciplinary co-design methodology – we included the end users from day one. More than 500 people worked with us in the requirement and development stages and gave valuable insights throughout the validation and testing phases. We learnt a lot about how to tailor XR for off-highway machine operators so the solutions we developed in the pilots would stand the test of real-life application,” says project coordinator Martijn Rooker, Senior Innovation Projects & Funding Manager at TTTECH.
The project’s results were for the most part incorporated directly into the industrial environments they were tested in – to further develop products and operator cabin setups, as well as being applied for future university research and curricula.
TTControl: THEIAXR pilot with Prinoth enhances smart operator assistance for snow groomers with XR
TTControl coordinated the THEIAXR project and led the consortium throughout its duration. Lorenzo Venditti, Senior Product Manager HMI at TTControl explains the role that such projects play: “Research and development are integral to our continuous efforts to test the use of new technologies. We explore how emerging technologies can realistically be adopted in industrial contexts. In THEIAXR we had the opportunity to assess how the interaction between operator and machine evolves in increasingly autonomous systems. The project showed that information derived from sensor fusion becomes meaningful only when it is effectively perceived and used by the operator, and how XR can play a key role in enabling this interaction.”
TTControl not only takes part in research initiatives but actively cooperates on application-oriented solutions with leading industrial partners in the Autonomous Operation Cluster (AOC). TTControl’s long-time customer Prinoth is part of the AOC and partnered with TTControl in the snow grooming pilot of THEIAXR.
Snow groomer operators need to work during the night and under all weather conditions. Due to the “whiteout” phenomenon, where visibility is severely limited, the operators rely on sensor and visualization assistance to drive and control their vehicles.
TTControl developed the visualization and interaction system infrastructure that was designed to host and integrate XR technologies within the mobile machinery’s cabins. Working closely with TTTECH, TTControl also played an active role in supporting the integration of these technologies into the project demonstrators, ensuring that the solutions were effectively embedded and operational within the real-world test scenarios.
TTControl contributed its computing platform FusionAI, which serves as the brain of the vehicle architecture, connecting data collection, processing, and output functions. As off-highway machinery often operates in environments with limited connectivity to the Internet and the cloud, an embedded edge computing platform enables on-site processing and direct communication of results to the operator in the cabin. FusionAI can drive up to four displays and provides multiple CAN and Ethernet interfaces for seamless integration into the cabin setup.
TTControl’s human machine interfaces (HMIs) also played a central role in THEIAXR, as they represented the primary interaction point for the operator. Within the project, TTControl integrated Vision X monitors as visualization units working in combination with the FusionAI outputs to enable the display of processed and fused sensor information.
One of the key learnings was that, with increasing system autonomy, the HMI becomes even more critical. Rather than adding more data, the challenge is to structure and prioritize information so that it can be effectively perceived and used by the operator. THEIAXR allowed TTControl to explore how XR can be integrated with traditional in-cabin monitors as part of a unified HMI concept, supporting situational awareness and transparency of autonomous system behavior.
Martin Kirchmair, Group R&D / Product Director at Prinoth, highlights the value the pilot brought: “We want to push the boundaries of what our machines can do. THEIAXR was a great opportunity to try new options for operator assistance and obstacle detection. We’ve been working with TTControl for many years on the electronic platforms for our fleet. The embedded computing platform we tested in the XR use case was easy to incorporate into the cabin and provided the performance and connections needed to roll out the new functions we tested.”
Thanks to real-time data collection and execution of applications with FusionAI, the operator can receive information, feedback, and alarms quickly directly in the cabin. Integrated AI acceleration ensures fast processing of camera and sensor data. This allows XR applications to run directly on the edge node in the machine. The setup for Prinoth was an individual solution that connected a range of sensors and actuators and fed information from AI-based object and person recognition back to the operator in an intuitive way.
Integrating an embedded edge computing platform means that the machine can operate independently of external computing resources, and less (if any) data needs to be transferred via Internet or cloud, thus increasing system safety and security. The platform solution is scalable and adaptable to different requirements and environments – from agriculture to construction or municipal vehicles. The key benefits are:
- Improved performance thanks to data being handled directly on the machine
- Cost-effectiveness, as multi-modal XR technologies can be integrated into a single computing platform
- A future-ready software framework that supports software-defined automation
The results from THEIAXR will be applied to improve next-generation operator-assistance systems through TTControl’s computing platforms and HMIs confirms Lorenzo Venditti: “The project and the demonstrator with Prinoth were valuable to us in further developing our computing platform solutions based on practical use cases. We learnt a lot from the end users and their feedback. Prinoth was TTControl’s first customer, and it was a pleasure to work together again in this research project. Close collaboration like this is essential for innovation and developments that fit the market’s needs and provide real benefits to the machine manufacturer as well as the operators.”
TTControl offers a portfolio of products and services for more autonomous off-highway machinery. Aside from computing platforms and HMIs as used in THEIAXR, these include end-to-end development services through TTControl’s Application Development Center (ADC). The dedicated Emerging Technologies team supports customers in integrating AI-powered perception, sensor fusion, and autonomous assistance into their machinery.
Learn more
- TTControl and Prinoth: Future-proof electronic control platforms for snow groomers
- CEO Talks: Automation to autonomy: From vision to execution (Guest: Martin Pottendorfer, TTControl)
- HORSCH uses TTControl's computing platform to further develop AI functionality in its spot spraying solution
- NXP Semiconductors: Sensor fusion for autonomous snow grooming with TTControl and Prinoth
- Related products and services:
- THEIA-XR: end of project video
- THEIA-XR website
