Petya Petrova - Fotolia


Real-time data analytics on IoT info starts with solid architecture

Data streaming into systems from IoT devices creates new potential for real-time analytics applications. But IoT data management basics need to be planned for first.

Internet-connected devices are rapidly being deployed across numerous IT networks, as organizations recognize the...

potential business value of analyzing the data they generate -- and doing so in real time as the information is transmitted.

Data sets are streaming into corporate systems from a plethora of internet of things (IoT) sources, including sensors, appliances, vehicles, smart personal devices, traffic monitors and industrial equipment. The increasing availability of IoT data streams coupled with the ability to ingest, process and analyze them sets the stage for equipping business operations with real-time data analytics capabilities -- among them streaming analytics, operational intelligence and continuous monitoring of data for security, auditing and regulatory compliance processes.

The information available on IoT includes machine-generated data that's automatically transmitted on a defined schedule. An example is a smart energy meter that sends power usage metrics to a central system every 15 minutes. In addition, there's what could be called automated human-generated data, which involves information that's created based on the actions of people -- for example, internet clickstream logs that track web activity, and event sequences within an online game that record the playing choices of gamers.

To take full advantage of the available data for real-time data analytics, an organization must adopt a strategy that encompasses the following four key aspects of IoT data management.

Ingestion. Effectively ingesting and analyzing a single data stream can inform the management of operational processes and point to ways to improve them; pulling in data from multiple streams is even more beneficial, enabling a holistic perspective on myriad business activities that executives are looking to monitor and analyze. A large set of IoT data sources provides a wealth of information that can be incorporated into predictive models to support internal decision-making.

But your data management architecture must be able to accommodate the volume and variety of the different types of data streams available for the taking, potentially including ones with different data formats that may need to be reconciled. You also could find yourself dealing with multiple streams from the same source or instances of the same type of stream from multiple sources, as well as data inputs from devices in different geographic locations.

Persistence. The volume of incoming IoT data may rapidly exceed your ability to process it all in real time; as a result, there may be a need to filters to limit the amount of data within each stream that will be processed and retained for analysis. That being said, don't mistake filtering for the sake of real-time analysis for throwing away data.

Filtering is a useful technique to speed automated streaming analytics, but your analytical models need to be developed upfront using historical data. And while your budget and the available capacity of persistent data storage it can cover are factors in determining whether to retain all of your IoT data or let some of it go, storage costs have declined significantly overall. If it makes sense financially, capture and store the entirety of each streamed data instance. The body of accumulated data will then form a system of record that can be explored by your data analysts and be used to build other predictive models in search of insights.

Analytics. Streaming analytics tools let users run predictive models against newly ingested IoT data in real or near real time. That can include analysis of trends and patterns in the data, as well as monitoring of "sentinel conditions" to trigger real-time notifications to business managers when predefined events transpire or problems with machinery are identified. To make those kinds of real-time data analytics processes feasible, your IoT architecture must incorporate analytics tools that can handle multiple data streams from different sources.

Communication. Monitoring IoT data to identify specific events or issues won't be useful if you can't send alerts to the right individuals so they can take action to address an emerging opportunity or problem. In addition to implementing mechanisms to enable real-time alerts and notifications, you'll need to set up processes for managing the groups of people who are to be notified (or the systems, in cases involving automated responses to alerts), as well as the content of the notifications, the means by which recipients are notified and confirmations that the alerts have been delivered.

When attempting to use IoT data from different sources to drive predictive analytics and prescriptive business behaviors, consider how these four foundational aspects of an IoT data management strategy should be designed and architected. Doing this before you deploy any technologies will simplify the implementation of real-time data analytics applications driven by IoT and keep your organization from drowning in a flood of data.

Next Steps

Growth of big data environments spurs on IoT analytics applications

Companies look to capitalize on the industrial IoT for data analytics

IoT 101: What the internet of things is and how the IoT works

Dig Deeper on Internet of Things (IoT) Analytics