How to Use Sensor Fusion to Improve Industry 4.0 Production Processes and Logistics

Sensor fusion combines data from multiple sensors to provide a more detailed and nuanced understanding of system operation or the environment. In many instances, the weakness of one sensor technology can be overcome by adding (fusing) information from a second sensor technology. Adding artificial intelligence (AI) and machine learning (ML) can enhance the power of sensor fusion.

There are several challenges to address when implementing sensor fusion. For example, it can be difficult to develop a balanced solution and not “favor” one of the technologies over the others. That can result in a lack of scalability and reduced performance. One way to address that challenge is to integrate multiple sensor technologies into a single package. Sensor fusion is not limited to the use of multiple discrete sensors.

Regardless of the level of sensor integration, adding AI or ML can improve performance, but training can be complex and time-consuming. Instead, designers can turn to self-training sensors with embedded AI and ML capabilities.

This article begins by reviewing a sensor fusion implementation using discrete sensors, a 32-bit MCU, and ML software. It then presents a series of integrated sensor fusion solutions and application examples in logistics facilities, data centers, process automation, material handling, and agricultural equipment.

It closes by looking at an integrated environmental sensor fusion solution with integrated AI software. Throughout the discussion, exemplary devices will be included from Renesas Electronics, Sensirion, TE Connectivity, ACEINNA, Bosch Sensortec and TDK InvenSense.

Designers can explore sensor fusion options using a reference design board from Renesas. The board is based on a 32-bit MCU with a 120-MHz Arm® Cortex®-M4 core, up to 2-MB code Flash memory, and 640-KB SRAM, plus numerous interface and connectivity options.

The related eval kit is optimized for multi-sensor and sensor fusion designs. It includes an air quality sensor, light sensor, temperature and humidity sensor, a 6-axis inertial measurement unit (IMU), a microphone, and Bluetooth Low Energy (BLE) connectivity (Figure 1). The reference design also includes an automated ML platform for edge devices and sensor fusion applications.

Figure 1: IoT sensor fusion eval and dev board with automated ML development software and BLE connectivity. (Image source: Renesas Electronics)

Stabilizing tilt sensors

Tilt sensors are specialized IMUs used in various applications, including agricultural machines, off-road vehicles, material handling, and heavy-duty construction equipment. Safety standards sometimes require tilt sensors to ensure safe operating environments. Tilt sensors can be assembled using several discrete devices, which can be complicated.

The core for most tilt sensor designs is a gyroscope sensor (gyro) that measures the angular velocity or rate of rotation around an axis. That’s great if the platform is in motion, but if it stops moving, say tilted at a 20-degree angle, the sensor output goes to zero. In addition, a gyro can experience significant drift over time, with the errors accumulating and eventually producing a no longer accurate or useful measurement.

To address the limitations of gyros, dynamic tilt sensor solutions add an accelerometer to measure motion. That can tell the system when it’s stopped moving and enable it to use the last output from the gyro to estimate the tilt angle. One final piece to the puzzle is a temperature sensor that compensates for the effects of varying temperatures on the gyro and accelerometer.

Kalman filters are often used for sensor fusion in tilt sensors. A standard Kalman filter based on linear quadratic estimation can be used if the sensors are being operated in a linear region of performance. Kalman filters can produce relatively accurate state estimates even in systems like tilt sensors with inherent uncertainty and accumulation errors.

Tilt sensors operating in a non-linear region can benefit from an extended Kalman filter that linearizes the estimates using the current mean value and covariance.

Tilt sensors like the AXISENSE-G-700 from TE Connectivity and the MTLT305D from ACEINNA have six degrees of freedom (6 DoF) motion sensing, three from the gyro and three from the accelerometer, and employ Kalman filtering techniques for sensor fusion (Figure 2).

Figure 2: The AXISENSE-G-700 tilt sensor fuses data from acceleration, rotation, and temperature sensors to provide accurate tilt information in dynamic environments. (Image source: TE Connectivity)

Fusing nine-in-one

While 6 DoF are enough in many cases, some motion-tracking applications like drones, vehicles, and virtual reality devices can benefit from the additional information provided by using 9 DoF.

The OPENIMU300RI module from ACEINNA is designed for use in 12 V and 24 V automotive, construction, and agricultural vehicles. In addition to a gyro and accelerometer, this IMU has a 3 DoF anisotropic magneto-resistive (AMR) magnetometer.

An ARM processor collects the sensor data and implements OpenIMU, an open-source stack for IMU, global positioning system (GPS), and inertial navigation system (INS) development. The stack includes a customizable Kalman filter for sensor fusion.

TDK InvenSense also offers a 9-Axis Motion-Tracking Device. Model ICM-20948 has an operating temperature range of -40°C to 85°C, making it suitable for various applications in challenging environments such as industrial automation and autonomous systems. It includes a microelectromechanical system (MEMS) based three-axis gyro, a MEMS-based three-axis accelerometer, and a MEMS-based three-axis magnetometer/compass.

In addition to the 9 DoF motion sensors, the ICM-20948 has independent analog-to-digital converters (ADCs) for each sensor, signal conditioning circuits, and a digital motion processor (DMP) (Figure 3).

Figure 3: This integrated sensor platform supports 9 DoF using a three-axis gyro and three-axis accelerometer (lift side) plus a three-axis magnetometer/compass (lower right). (Image source: TDK InvenSense)

Some ICM-20948 details include:

Three independent vibrational MEMS rate gyros. If the gyros are rotated about any of the three axes, the Coriolis Effect causes a vibration detected by a capacitive pickup. The output from the pickup is processed to produce a voltage that’s proportional to the angular rate.

The 3-axis MEMS accelerometer has separate masses for each axis. Acceleration along an axis displaces the corresponding mass, which a capacitive pickup detects. When the ICM-20948 is placed on a flat surface, it will measure 0g on the X- and Y-axes and +1g on the Z-axis.

The magnetometer is based on Hall sensor technology. It detects terrestrial magnetism in the X-, Y-, and Z-axes. Sensor output is generated with a sensor driver circuit, an amplifier, a 16-bit ADC, and an arithmetic circuit for processing the resulting signal. Each axis has a full-scale range of ±4900 µT.

The DMP in the ICM-20948 is a differentiator. Some of its features and benefits include:

• Offloading computation of motion processing algorithms from the host processor minimizes power consumption and simplifies timing and the software architecture. The DMP ensures that motion processing algorithms can be run at a high rate, around 200 Hz, to provide accurate results with low latency. Operation at 200 Hz is recommended, even if the application updates much slower, like 5 Hz. Decoupling the DMP processing rate from the application update rate ensures more robust system performance.
• The DMP enables ultra-low power run-time and background calibration of the sensors. Calibration is needed to maintain optimal performance of individual sensors and sensor fusion processes over the device's lifetime.
• The DMP simplifies the software architecture and speeds software development resulting in quicker time to market.

Integrated environmental sensors

Environmental monitoring is essential in food processing and storage, chemical plants, logistics operations, data centers, greenhouse crop production, heating, ventilation, and air conditioning (HVAC) systems, and other areas. Relative humidity (RH) and temperature measurements can be fused to calculate the dew point.

Sensirion's SHTC3 series are digital humidity and temperature sensors optimized for battery-driven applications on the edge and in high-volume consumer electronics. The CMOS sensor platform includes a capacitive humidity sensor, a bandgap temperature sensor, analog and digital signal processing, A/D converter, calibration data memory, and an I²C fast-mode communication interface.

The small 2 x 2 x 0.75 mm DFN package supports space-constrained applications. The broad supply voltage of 1.62 V to 3.6 V and an energy budget below 1 μJ per measurement make the SHTC3 suited to battery-powered mobile or wireless devices (Figure 4). For example, part number SHTC3-TR-10KS is delivered in quantities of 10,000 on Digi-Reel, tape and reel, or cut tape. Designers can use the SHTC3 eval board to speed system development.

Figure 4: This environmental monitoring device includes digital humidity and temperature sensors. (Image source: Sensirion)

Adding barometric pressure

Context and location awareness are increasingly important in home automation controls, HVAC systems, fitness equipment, and indoor navigation applications. The designs of those systems can benefit from using the BME280 integrated environmental unit from Bosch Sensortec that adds a barometric pressure sensor along with humidity and temperature sensors.

The sensors are low noise designs that provide high accuracy and resolution. The pressure sensor measures absolute barometric pressure. The integrated temperature is optimized to work with the humidity sensor to determine RH and dew point. It’s also used to provide temperature compensation for the barometer. A dev board is available to speed up the design and system integration process.

AI for environmental sensing

Bosch Sensortec also offers a 4-in-1 Environmental Sensor with embedded AI. The BME688 includes a gas sensor and high-linearity and high-accuracy pressure, humidity, and temperature sensors. It’s delivered in a rugged 3.0 mm x 3.0 mm x 0.9 mm package suited for mobile and other space-constrained applications (Figure 5).

Figure 5: Bosch Sensortec's BME688 includes a gas sensor plus pressure, humidity, and temperature sensors, all supported with integrated AI. (Image source: Bosch Sensortec)

The gas sensor can detect volatile organic compounds (VOCs), volatile sulfur compounds (VSCs), and other gases like carbon monoxide and hydrogen in the parts per billion (ppb) range. The BME688 includes a gas scanner function that can be customized for sensitivity, selectivity, data rate, and power consumption.

The BME AI-Studio software also optimizes the gas sensor for other gas mixtures and applications. The BME688 eval board can be configured with the BME AI-Studio software. BME AI-Studio supports sensor configuration, data analysis and labeling, training, and optimization of application solutions for factories, logistics facilities, smart home, and IoT devices.

Sampling gases and training the system in the field instead of the lab enables the design of more realistic algorithms that perform better and provide higher levels of reliability under actual operating conditions. By taking advantage of the BME688’s ability to simultaneously measure humidity, temperature, and barometric pressure, in addition to gases, more comprehensive and accurate AI models can be developed.

Conclusion

Sensor fusion systems for Industry 4.0, logistics, and other applications can be developed using a series of discrete sensors or an integrated solution that includes multiple sensors in a single package. Integrated devices can produce smaller and lower-power solutions for mobile and edge applications. Whether using discrete sensors or an integrated sensor suite, performance can be enhanced by adding AI and ML.