Why massive IIoT data volume is spurring time series database growth
While they’ve been around for a while now, time series databases capable of handling large-scale data in real time are rapidly becoming critical for organizations that depend on industrial IoT technologies. Beginning with use cases such as sensor data collection or the monitoring of infrastructure, TSDBs have now expanded to deliver organizational advantages across nearly any field using IIoT capabilities.
Certainly, the major cloud providers have been spurred to offer services in line with this trend, with AWS introducing Amazon Timestream and Azure now offering its Time Series Insights PaaS. The days of storing large volumes of sensor information in traditional databases not specifically intended for time series data are ending, as cloud cost-savings and technical features, like query performance and scalability, favor a transition to TSDBs.
At the same time, new tools are equipping industrial businesses with new capabilities for collecting infrastructural time series data — Prometheus is a strong example. The increase in the benefits derived from time series data will elevate this information as a competitive differentiator among IIoT-empowered enterprises, within which the advantage will go to those organizations best able to collect time series data at scale and glean the most value from that data. In this environment, industrial businesses that are simply storing and plotting data points won’t be able to keep pace with competitors equipped to more nimbly harness massive data and act upon key insights.
The coming TSDB divide
This transition toward widespread adoption will lead to a stark division in how TSDBs are put to use — and which specific TSDBs are appropriate to different use cases — with two distinct categories emerging:
1. Smaller, more traditional IoT use cases (mostly operational metrics)
The first category covers use cases considered more traditional, which utilize just tens or hundreds of metrics or IoT sensors, and require real-time write abilities but no complex queries and little in the way of special requirements for integration. These use cases will produce data volumes totaling less than a terabyte and will usually be internal IT projects that are non-mission critical in nature. Appropriate TSDBs for use cases in this category include Prometheus and InfluxDB.
2. Hyperscale IIoT application
This category includes industrial time series applications utilizing hundreds or even hundreds of thousands of IoT sensors or metrics. These applications call for deep integration between IT and operational technology, metrics logs and search functionality, and the ability to perform real-time queries even with highly concurrent load — for example, thousands of queries per second or tens of thousands of concurrent connections. Doing so makes it possible to use interactive dashboards, alerts, stream processing, machine learning and more.
In these use cases, data volumes can stretch from gigabytes up into the hundreds of terabytes and represent mission-critical systems, such as platforms offering real-time, data-driven decision-making — which can have a transformative effect on an IIoT-dependent business’ capabilities. The scale of these use cases requires enterprises to use TSDBs — open source CrateDB being one — up to the task, such as those offered by or cloud service providers or specialized DBaaS providers.
How cloud service providers and database-as-a-service providers are adapting to these shifts
DBaaS providers — especially cloud service providers — are thriving in this environment, where industrial organizations need to adopt new architectures and see the wisdom in enlisting experts to do so. In fact, AWS and Azure by themselves are responsible for two-thirds of last year’s total database market growth.
However, it’s important for industrial organizations to keep an eye on how providers’ pricing models evolve as IIoT implementations thrive and as hyperscale TSDB use cases become more prevalent. Currently, provider cost structures often assume a limited scale — for example, AWS Timestream offers $350 per month pricing, but with a limit of 100 queries on the data set per month.
For enterprises with industrial platforms that must execute thousands of queries each second, this means that costs quickly become unpredictable, a fact that market pricing will adapt to address. Additionally, watch for DBaaS providers to support enterprises in adopting hybrid cloud strategies that offer versatility and resilience while avoiding future lock-in risks. By designing database architecture to utilize any cloud, and to easily scale so that the same code is used and the same features are available for small-scale IIoT use cases as for massive IIoT-based cloud deployments, DBaaS providers can simplify decision-making and deliver functionality that empowers industrial organizations with competitive advantages.
All IoT Agenda network contributors are responsible for the content and accuracy of their posts. Opinions are of the writers and do not necessarily convey the thoughts of IoT Agenda.