freshidea - Fotolia

IoT messaging: The MQTT protocol is stepping up to the plate

The MQTT protocol is more than 15 years old, but its prevalence is now increasing as the Internet of Things takes hold.

According to one of its creators, the MQTT protocol hasn't worn out its usefulness, even though it's been kicking around since the pre-dawn of the Internet of Things.

The open publish/subscribe messaging transport protocol was invented by Andy Stanford-Clark of IBM and Arlen Nipper currently of Cirrus Link Solutions. The need for MQTT arose when the pair was helping oil and gas companies find an effective data transport protocol in the mid-1990s. While a slew of other protocols were available, most required a switch statement to process the data which, while doable with a static number of devices, would be increasingly challenging as more devices came onto the scene.

To solve the problem, Stanford-Clark and Nipper released the data agnostic MQTT protocol in 1999. At the time, MQTT was short for MQ Telemetry Transport, the MQ referring to IBM's MQ Series; the acronym later dropped its spelled out version.

MQTT admittedly took a while to catch on. Though part of the public domain since 1998, it was often viewed as proprietary since only IBM was using it. After Roger Light asked Stanford-Clark about MQTT's status in 2009 and found out it wasn't proprietary, he went home and created the open source MQTT broker Mosquitto, which was released weeks later.

Various MQTT implementations soon popularized the protocol, and three years after its public release it was announced MQTT would be standardized. MQTT was officially approved as an OASIS standard on Oct. 28, 2015, further fueling its use among large organizations. MQTT was accepted as an ISO standard at the end of January 2016; IEC 20922 is expected in 2017.

What is the MQTT protocol exactly?

The pub/sub protocol has been broadly used in M2M environments, and now has had new life breathed into it thanks to IoT. MQTT is preferred over other protocols (such as HTTP) and traditional client/server exchanges due to its minimal packet overhead.

Just like the postal service, MQTT doesn't care what you send. Whatever you put in the letter box, it transports it for you, delivers it to the destination and presumes the person at the other end knows how to read it.
Andy Stanford-Clarkdistinguished engineer, IBM

Stanford-Clark attributes MQTT's success to its simplicity. Unlike other protocols, MQTT does not require standard formatting for data such as temperatures or time stamps; organizations can design a payload for messages to customize it to their own needs.

"Just like the postal service," Stanford-Clark said, "(MQTT) doesn't care what you send. Whatever you put in the letter box, it transports it for you, delivers it to the destination and presumes the person at the other end knows how to read it."

MQTT is also popular as its clients and subscribers can be any array of devices or systems that has an MQTT library running and connects to an MQTT broker. Client libraries are available in a number of programming languages including Android, Arduino, C, C++, iOS and JavaScript, or organizations may write their own MQTT library.

MQTT's low bandwidth makes it ideal for constrained devices and unreliable networks, though it is often also used on reliable networks to preserve bandwidth. MQTT enables bidirectional communication, even behind NAT routers. It uses a TCP connection initiated by the client to connect client and broker; maintenance of this connection generates a 2 byte overhead.

The MQTT protocol is used today throughout a variety of industries and IoT environments, including smart homes, manufacturing and energy. After the launch of Facebook Messenger, which leverages MQTT to not only save battery life but also deliver messages in "the hundreds of milliseconds," many other applications followed suit. Karma Mobility's mobile hotspot connects to its back-end infrastructure using MQTT, and Amazon Web Services became the third major platform provider (after IBM and Microsoft) to announce MQTT as its primary transport protocol.

The role of the MQTT broker

"The MQTT broker is the heart of each MQTT deployment," explained Christian Götz, CEO at dc-square GmbH, the HiveMQ Company. "It is the connecting link between physical devices or applications and enterprise systems."

MQTT uses a broker to facilitate messages between those sending messages (clients) and those receiving messages (subscribers) based on the message's subject. MQTT brokers collect messages from clients and use a hierarchical string to filter and route them to the appropriate subscribers. Some broker service providers also offer content-based filtering, separating and routing message based on content rather than subject.

Brokers are responsible for subscriptions, persistent sessions, missed messages and overall security, including authentication and authorization. Enterprises seeking an MQTT broker should evaluate broker scalability, integration and monitoring capabilities, and failure resistance.

MQTT drawbacks

According to Stanford-Clark, security was consciously left out of the protocol initially because he and Nipper knew security mechanisms could be wrapped around MQTT to boost security. Also, at the time, Stanford-Clark said information sent via MQTT, such as wind speed data from a weather station, wasn't particularly in need of securing.

Michael Cobb, CISSP-ISSAP, warned that times have changed, and organizations using MQTT must take this into account.

"My main concern is history will repeat itself and vendors/developers will rush to be first to market, which normally means no attention paid to security," Cobb said. "With IoT and MQTT, security requirements will vary greatly dependent on the use case, but this is a new discipline for everyone, there are no clear best practices other than those that can be applied from general network security."

"As we know, the same old flaws/errors keep turning up even in the best protected networks," Cobb said. "So my advice is that project managers should look at how to solve the security issues for their IoT products before development begins so security is baked in and not applied later as a sugar coating to placate users who have fallen victim to a hacker attack."

Brokers can also make or break the MQTT deal.

"The architecture of MQTT is based on the broker being available," Götz said. "Therefore, any MQTT broker must be highly available."

Götz also noted security concerns, but said additional measures -- such as combining MQTT with state-of-the-art security standards like TLS and using X.509 certificates for client authentication -- can help improve security. Such measures are generally the responsibility of the broker, so enterprises should vet choices properly.

Subject-based filtering is another challenging aspect of MQTT, it must be properly laid out beforehand. Both publisher and subscriber must be aware of the topics to ensure delivery, but even so, if no devices subscribe to a particular subject, the message could have no recipients.

Where the MQTT protocol is headed in the future

"The unique feature set is mainly responsible for MQTT becoming one of the de-facto IoT protocols," Götz said. Besides its low bandwidth consumption, bidirectional communication and ease of use, MQTT also has "built-in QoS levels which make application level logic of resending obsolete and saves time in development of applications."

"It's stood the test of time," Stanford-Clark said. "It was 10 years before we considered making any changes at all to MQTT, so that version, which we published in 1998, was perfectly fit for purpose for a good 10 or more years."

MQTT version 5, expected in 2017, will include some updates, however.

"The committee is deciding to add some features (to version 5) that have been proposed which are not efficiencies," Stanford-Clark said. "But there are some things which a high-scale, high-throughput, hundreds-of-millions-of-devices type world that we're anticipating in the near future needs. There are some parts of MQTT which in 1998 we didn't even think about, we were still thinking about tens of thousands of devices connecting into a broker, not a few hundreds of millions of devices connecting into a broker; some of the design implications MQTT puts on you have been challenged."

For example, Stanford-Clark said, "Message expiry. At the moment in MQTT there is no message expiry, so if you put a message in a broker and then forget to collect it or no one ever comes to pick it up, it stays there forever, whereas if you had a message expiry policy you have, say, three weeks; after three weeks it evaporates so you wouldn't slowly clog up your broker with a crust of old messages that no one had thought to receive."

So will MQTT's resurgence last the tests of time?

"This is just the beginning," Götz said. "We will see more MQTT-enabled systems in the next few years. Also, a lot of companies that rely on MQTT today or have in the past are in for the long run; they have chosen MQTT and will use it at least for the next five to 10 years."

"I'm pretty certain MQTT's not going away any time soon," Stanford-Clark said. "Version 5 will make it fit for the next 20 or 30 years."

Next Steps

Prepare as the enterprise IoT wave comes in

So what is the Internet of Things exactly?

Don't let predatory practices hinder IoT

Dig Deeper on Internet of Things (IoT) Standards and Certifications