Machine Learning Powered Home Scenes: A Blueprint for Intelligent Home Automation

This is a summary of an article from a five-part series about the technologies that enable IoT. The full article can be viewed at wevolver.com.

For over two decades, smart homes and home automation applications have advanced at a rapid pace. Early smart homes comprised a limited number of connected devices and offered very basic automation functionalities, which were typically limited to controlling the status of smart devices. During the past decade, home automation functionalities have proliferated and evolved in terms of intelligence and sophistication. Specifically, smart home platforms like Amazon Echo, Google Home, and Apple’s Home Pod enable integrated and intelligent control of a larger number of devices. The latter can be configured in many ways to match the end-users preferences and needs.

Introducing Smart Home Scenes

The multitude of home automation devices and functionalities in modern smart homes has given rise to the concept of “scenes”. Scenes enable smart home users to control multiple devices based on one or more scripts of well-defined actions. The latter are scheduled and executed in line with the end-users preferences. In practice, scenes consist of triggers and actions. Triggers are events that initiate configuration and control actions. For example, configuration actions can be triggered when the value of a sensor exceeds a threshold or based on the manual push of a button. Actions alter the status of one or more devices, which subsequently changes the state of the smart home.  The triggers and actions that comprise a scene are stored in the smart home platform as a configuration option. This option can be conveniently activated or deactivated at the end-users will. Smart home users can therefore configure multiple scenes and use them at different times.

Overall, scenes facilitate homeowners to control their smart devices flexibly and automatically. Using scenes, homeowners save precious time. Specifically, they no longer need to remember the different configuration options of their devices, allowing them to spend less or no time in implementing a series of configuration and control actions to ensure that their home environment matches their preferences. Furthermore, scenes enable homeowners to control their home using their most preferred modality, including remote controls, touch panels, and smartphone apps.

Smart Home Scenes: Technology Enablers and Trends

The implementation and deployment scenes are enabled by cutting-edge smart home technologies such as intelligent sensors, cloud computing, and the Internet of Things (IoT). In particular:

• Intelligent Sensors and Actuators: Scenes leverage intelligent semiconductors and microsystems, which are configured remotely based on popular protocols like Bluetooth, WiFi, Zigbee, and KNX. Intelligent semiconductors are of two main types: Sensors that perceive the smart home environment (e.g., a room’s occupancy patterns) and Actuators that implement automatic control functionalities (e.g., light control).

• IoT Controllers: Most scenes are configured, stored, deployed, and executed by IoT controllers i.e., gateway devices that control sensors and actuators from a single-entry point. IoT controllers support one or more networking protocols that facilitate their seamless communication with sensors and actuators.

• Cloud Computing: Modern smart home platforms and their IoT controllers are integrated with cloud computing infrastructures. Cloud integration enables smart home applications to benefit from the scalability, capacity, elasticity, and quality of service of cloud computing. For instance, the cloud facilitates the collection, management, and analysis of large volumes of data about the activity of the inhabitants. Likewise, cloud computing enables smart home applications to access storage and computing resources in flexible, scalable, and cost-effective ways.

• Edge Computing: In recent years, there is a shift of smart home data from the cloud to the edge of the network, i.e., close to end-users. Many data processing and data analytics applications are deployed close to the end-users. This reduces application latency as processing takes place in the local network.  Moreover, it economizes network bandwidth and boosts data protection as most data need not travel to the cloud. In most cases, edge computing is deployed in conjunction with cloud computing, i.e., real-time functionalities are deployed at the edge, while functionalities that ask for many data points and heavier processing are implemented in the cloud.

These technologies are the main technology enablers of smart home applications and scenes. Their evolution is driving improvements in automation, intelligence, and sustainability. Some of the top trends include:

• Tiny Sensors

• Energy Efficiency

• Cyber-Physical Systems and Robotic Innovations

• Strong Cybersecurity and Reliability

• Increased Automation and Real-Time Operations

Smarter Scenes with Machine Learning and Artificial Intelligence

The blending of ML functions into smart home scenes is the next evolutionary step in home automation. Most of the state-of-the-art scenes are based on simple if-then-else rules over triggers and actions. This deterministic logic boosts automation, yet it provides very limited intelligence.  For example, simple rules do not consider the evolution of users’ habits and behaviors. ML alleviates these limitations by enabling scenes that learn the residents’ behavioral patterns and use them to optimize smart home operations. In this direction, ML algorithms can be trained using past data about the users’ activities in their home environment. Leveraging such algorithms, scenes can be enhanced in terms of dynamicity and intelligence. Specifically, rather than always applying the same configuration actions during a given time interval, ML-enabled scenes can dynamically change the actions in-line with the evolution of the residents’ behavior. Moreover, the use of ML can lift the limitations of choosing between a few pre-configured scenes for different times of the day (e.g., morning, afternoon, evening). This is because ML-based scenes can continually and dynamically adapt to the users’ habits and behavior, even when the latter changes several times during the day.

Prominent examples of user-centric ML-based scenes include:

• Personalized Automated Home Ambience – Learning Users’ Preferences: In this case, the scenes automatically learn the users’ preferences about the ambient environment, i.e., about the lighting system, the operation of the air-conditioning, the configuration of the music player, the personal assistance services, the shades, the doors and more. Accordingly, they dynamically personalize the ambient environment to the users’ preferences.

• Occupancy Monitoring for Cost and Carbon Footprint Reduction: This type of scene leverages machine learning and computer vision recognition systems to extract occupancy patterns in the different rooms of the residential environment. Leveraging the identified patterns, the scenes configure the home’s operations (e.g., lighting, air conditioning) in ways that minimize energy costs and optimize CO2 emissions.

• Intelligent Recommendations of “Optimal” Scenes – Learning Scenes Dynamically: Leveraging past data about the users’ in-home behavior and the operation of the different devices, it is possible to construct scenes (i.e., smart home configurations) that economize on costs and optimize environmental performance. These configurations can be recommended to the end-users. Following the end-users approval, they can be stored and deployed in the smart home. In this case, ML enables the dynamic construction of “optimal” scenes rather than relying on pre-configured ones.   

Summary

The implementation of ML-based scenes hinges on the collection and processing of large volumes of data about operations, triggers, and actions, including users’ activities and behaviors within the residency. These data can enable the training of different types of ML algorithms, including traditional ML techniques (e.g., Decision Trees, Random Forests and SVM (Support Vector Machines)) and Deep Learning algorithms (e.g., Artificial Neural Networks (ANN) and Recurrent Neural Networks (RNN)). In cases where very large amounts of past data are available, ML techniques provide better performance and accuracy in understanding behavioral patterns and tailoring scenes’ configurations accordingly.

Scenes are nowadays an integral element of smart home intelligence. They boost home automation in ways that provide convenience to end-users and help them apply their desired configuration actions faster. Nevertheless, state-of-the-art scenes are quite simple and static. In this context, the emerging wave of ML-enabled scenes is set to disrupt the smart home applications landscape by enabling the development of more dynamic and intelligent scenes. ML is the next evolutionary step in smart home scenes. It will act as a catalyst for realizing the shift from existing rule-based scenes (Scenes 1.0) to the next generation of dynamic and highly personalized scenes (Scenes 2.0). Smart home deployers must therefore plan for integrating ML functions into their solutions to provide consumers with new levels of comfort and personalization.

The full version of this article can be viewed at wevolver.com.