MQTT (MQ Telemetry Transport)

Contributor(s): Peter Waher

MQTT (MQ Telemetry Transport) is a lightweight messaging protocol that provides resource-constrained network clients with a simple way to distribute telemetry information. The protocol, which uses a publish/subscribe communication pattern, is used for machine-to-machine (M2M) communication and plays an important role in the internet of things (IoT).

MQTT enables resource-constrained IoT devices to send, or publish, information about a given topic to a server that functions as an MQTT message broker. The broker then pushes the information out to those clients that have previously subscribed to the client's topic. To a human, a topic looks like a hierarchical file path. Clients can subscribe to a specific level of a topic's hierarchy or use a wild-card character to subscribe to multiple levels.

The MQTT protocol is a good choice for wireless networks that experience varying levels of latency due to occasional bandwidth constraints or unreliable connections. Should the connection from a subscribing client to a broker get broken, the broker will buffer messages and push them out to the subscriber when it is back online. Should the connection from the publishing client to the broker be disconnected without notice, the broker can close the connection and send subscribers a cached message with instructions from the publisher.

An example of MQTT's publish/subscribe model
MQTT's publish/subscribe model

History of the MQTT protocol

MQTT was created by Dr. Andy Stanford-Clark of IBM and Arlen Nipper of Arcom -- now Eurotech -- in 1999 as a cost-effective, reliable way to connect monitoring devices used in the oil and gas industries to remote enterprise servers. When challenged with finding a way to push data from pipeline sensors in the desert to off-site SCADA (supervisory control and data acquisition) systems, they decided upon a TCP/IP-based publish/subscribe topology that would be event-driven to keep satellite link transmission costs down.

Although MQTT is still closely associated with IBM, it is now an open protocol that is overseen by the Organization for the Advancement of Structured Information Standards (OASIS).

Though the name suggests it, MQTT is not part of the original IBM MQ series; however, as of version 7.1, it is available in WebSphere MQ. MQTT was previously known as the SCADA protocol, MQ Integrator SCADA Device Protocol (MQIsdp) and WebSphere MQTT (WMQTT), although all of these variations have fallen out of use.

How MQTT works

An MQTT session is divided into four stages: connection, authentication, communication and termination. A client starts by creating a TCP/IP connection to the broker by using either a standard port or a custom port defined by the broker's operators. When creating the connection, it is important to recognize that the server might continue an old session if it is provided with a reused client identity.

The standard ports are 1883 for non-encrypted communication and 8883 for encrypted communication using SSL/TLS. During the SSL/TLS handshake, the client validates the server certificate to authenticate the server. The client may also provide a client certificate to the broker during the handshake, which the broker can use to authenticate the client. While not specifically part of the MQTT specification, it has become customary for brokers to support client authentication with SSL/TLS client-side certificates.

Because the MQTT protocol aims to be a protocol for resource-constrained and IoT devices, SSL/TLS might not always be an option and, in some cases, might not be desired. In such cases, authentication is presented as a clear-text username and password that is sent by the client to the server as part of the CONNECT/CONNACK packet sequence. Some brokers, especially open brokers published on the internet, will accept anonymous clients. In such cases, the username and password are simply left blank.

MQTT is called a lightweight protocol because all its messages have a small code footprint. Each message consists of a fixed header -- 2 bytes -- an optional variable header, a message payload that is limited to 256 MB of information and a quality of service (QoS) level.

The three different quality of service levels determine how the content is managed by the MQTT protocol. Although higher levels of QoS are more reliable, they have more latency and bandwidth requirements, so subscribing clients can specify the highest QoS level they would like to receive.

The simplest QoS level is unacknowledged service. This QoS level uses a PUBLISH packet sequence; the publisher sends a message to the broker one time and the broker passes the message to subscribers one time. There is no mechanism in place to make sure the message has been received correctly, and the broker does not save the message. This QoS level may also be referred to as at most once, QoS0, or fire and forget.

An explanation of MQTT's QoS levels
MQTT quality of service levels

The second QoS level is acknowledged service. This QoS level uses a PUBLISH/PUBACK packet sequence between the publisher and its broker, as well as between the broker and subscribers. An acknowledgement packet verifies that content has been received and a retry mechanism will send the original content again if an acknowledgement is not received in a timely manner. This may result in the subscriber receiving multiple copies of the same message. This QoS level may also be referred to as at least once or QoS1.

The third QoS level is assured service. This QoS level delivers the message with two pairs of packets. The first pair is called PUBLISH/PUBREC, and the second pair is called PUBREL/PUBCOMP. The two pairs ensure that, regardless of the number of retries, the message will only be delivered once. This QoS level may also be referred to as exactly once or QoS2.

During the communication phase, a client can perform publish, subscribe, unsubscribe and ping operations. The publish operation sends a binary block of data -- the content -- to a topic that is defined by the publisher.

MQTT supports message BLOBS up to 256 MB in size. The format of the content is application-specific. Topic subscriptions are made using a SUBSCRIBE/SUBACK packet pair. Unsubscription is similarly performed using an UNSUBSCRIBE/UNSUBACK packet pair.

Topic strings form a natural topic tree with the use of a special delimiter character, the forward slash (/). A client can subscribe to -- and unsubscribe from -- entire branches in the topic tree with the use of special wild-card characters. There are two wild-card characters: a single-level wild-card character, the plus character (+); and a multilevel wild-card character, the hash character (#). A special topic character, the dollar character ($), excludes a topic from any root wild-card subscriptions. Typically, the $ is used to transport server-specific or system messages.

MQTT message descriptions
MQTT protocol messages

The fourth operation a client can perform during the communication phase is to ping the broker server using a PINGREQ/PINGRESP packet sequence, which roughly translates to ARE YOU ALIVE/YES I AM ALIVE. This operation has no other function than to maintain a live connection and ensure the TCP connection has not been shut down by a gateway or router.

When a publisher or subscriber wants to terminate an MQTT session, it sends a DISCONNECT message to the broker, and then closes the connection. This is called a graceful shutdown because it gives the client the ability to easily reconnect by providing its client identity and resuming where it left off.

Should the disconnect happen suddenly without time for a publisher to send a DISCONNECT message, the broker may send subscribers a message from the publisher that the broker has previously cached. The message, which is called a last will and testament, provides subscribers with instructions for what to do if the publisher dies unexpectedly.

MQTT protocol applications and use cases

Facebook currently uses MQTT for their messenger app, not only because the protocol conserves battery power during mobile phone-to-phone messaging, but also because, in spite of inconsistent internet connections across the globe, the protocol enables messages to be delivered efficiently in milliseconds.

Examples of MQTT use cases
MQTT vertical applications

Most major cloud services providers, including AWS, Google Cloud, IBM Bluemix and Microsoft Azure, support MQTT, as do the Carriots, Evrything and ThingWorx IoT platforms.

MQTT is well-suited to applications using M2M and IoT devices for real-time analytics, preventative maintenance and monitoring, among other uses, in environments such as smart homes, healthcare, logistics, industry and manufacturing.

Competing protocols

Other transfer protocols under consideration for IoT devices with constrained resources include the Constrained Application Protocol (CoAP), which uses a request/response communication pattern, and the Advanced Message Queuing Protocol (AMQP), which, like MQTT, uses a publish/subscribe communication pattern.

MQTT challenges: Security, interoperability and authentication

Because the MQTT protocol was not designed with security in mind, the protocol has traditionally been used in secure back-end networks for application-specific purposes. MQTT's topic structure can easily form a huge tree, and there's no clear way to divide a tree into smaller logical domains that can be federated. This makes it difficult to create a globally scalable MQTT network because, as the size of the topic tree grows, the complexity increases.

Another negative aspect of MQTT is its lack of interoperability. Because message payloads are binary, with no information as to how they are encoded, problems can arise -- especially in open architectures where different applications from different manufacturers are supposed to work seamlessly with each other.

As touched upon previously, MQTT has minimal authentication features built into the protocol. Username and passwords are sent in clear text and any form of secure use of MQTT must employ SSL/TLS, which, unfortunately, is not a lightweight protocol.

Authenticating clients with client-side certificates is not a simple process, and there's no way in MQTT, except using proprietary out-of-band means, to control who owns a topic and who can publish information on it. This makes it very easy to inject harmful messages, either willfully or by mistake, into the network.

Furthermore, there's no way for the message receiver to know who sent the original message unless that information is contained in the actual message. Security features that have to be implemented on top of MQTT in a proprietary fashion increase the code footprint and make implementations more difficult.

Recent updates and the future of MQTT

In spite of its challenges, many experts believe that MQTT will play an important role in the internet of things, facilitating such things as inventory tracking, automotive telematics, resource monitoring and medical body area networks (MBANs). The protocol is continuously improving and now supports WebSockets, another protocol that enables two-way communication between clients and brokers in real time.

MQTT was officially approved as an OASIS standard on Oct. 28, 2015. At the end of January 2016, it was accepted as an ISO standard, with ISO/IEC 20922 announced in June 2016.

This was last updated in February 2018

Continue Reading About MQTT (MQ Telemetry Transport)

Dig Deeper on Internet of Things (IoT) Standards and Certifications

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

Why do messaging protocols play such an important part in the Internet of Things?
Why? Simply speaking, because it's the web of connections that made Internet to be. All these small devices need to transfer and receive data, most often - wirelessly. The messaging protocol must be light on bandwidth and speed while very robust to survive latencies.
What about Wi-Fi HaLow? Won't Wi-Fi offer more security than any of the various messaging protocols?

We (humans) make funny sounds that sometimes make words being combined certain way. Words sometimes come together as expressions and sentences. Sentences may make more or less sense. Sometimes they even come together as a whole speech.

IT protocols are funny this way, too. They need to be stacked.

For example, Web page protocol (HTTP) is a layer above TCP (Transmission Control Protocol) which is on top of IP (Internet Protocol). And there are protocols in between, like UDP.

Similar with wireless communications.

First it's an emission of energy using some medium. Typically it's electromagnetic waves at some frequency and amplitude.

There must be a protocol for frequency range, strength of signal, and other technical parameters. WiFi takes care of this level.

On top of that there are protocols responsible for data transfer, session control, encryption, and so on.

On top of that we have the same HTTP so you can see the same web pages the same way on wired and wireless.

And if the goal is to transport data like telemetry instead of web pages we need to have specialized protocols like MQTT instead of HTTP/TCP/IP. Same WiFi may stay beneath.

On the other hand, MQTT was originally created for "cost-effective, reliable way to connect monitoring devices used in the oil and gas industries". We may imply that field conditions might be too different from office conditions. Physical characteristics of firm rock may require different kind of WiFi even to penetrate the matter. Distance might be also a bit of couple kilometers greater than in the office so greater strength of the signal might be needed to reach the receiver.
Well, it's like the tootsie pop. You need a way to communicate the chewy center that everyone wants. Seriously, IoT wouldn't be a think if what's going on in them wasn't transmittable in some form. The web has pretty much been one big messaging platform from the get go: SMTP, HTTP(s), SSL, TCP/IP, UDP, FTP, sFTP, TFTP, etc. All are protocols designed to handle data transmission. The problem with IoT is that data possibly could get misused, and is probably more susceptible to some of the security issues.

I like that this article talks about all the handshaking that goes on, so many times that gets taken for granted on the web today, but it is really key for the net to work.
Sometimes when I'm reading/learning about the protocols that are supporting IoT, I think I've time traveled back to 1984 when TCP/IP was battling it out with ISO/OSI to become "the standard." It's interesting and it's exciting -- because this time at the rodeo, security is a major star in the lineup and not just watching from the side.
Well, because IoT is all about devices or "things" being able to communicate with each other, and they communicate via messaging protocols. With the volume of devices it's important that they are able to communicate fast & efficiently.
MQTT uses TCP/IP protocol then how it is lightweight protocol. It has header of TCP/IP and MQTT.
definitely Iam superb excited with this communities thats why I supposted to joininig.Anyhow. phenomenal tolks


File Extensions and File Formats