The Internet of Things involves adding digital sensors and networking technologies to the devices and systems that we use every day in the analog world. I have worked in a project of Smart metering in Berlin in my stint in a Utilities company.
Smart meters as the name suggest are devices who automatically reads the consumed units from the electricity meters installed in home and offices. IoT is not only limited to smart meters but to all the things around you like vehicles, coffee machine, echo smart speakers etc.
Now as this data is generated, it has to be captured. According to Gartner, in 2020 , it is expected to have 20 billion Internet of Things devices around the globe and steadily rising. I have divided the process in 5 parts.
Capture Data
Ingest the Data
Pipeline
Storage & Analytics
Presentation & Application
Capture Data
The data is being generated by IoT devices. These devices are connected to the IoT gateway via bluetooth, wifi,cellular etc. One or many IoT devices can be connected to a single IoT Gateway.
Now the question may arise, why does one need IoT Gateway and not directly to the Cloud. Well IoT Gateway is a solution for enabling communication usually between device to device connection and device to cloud connection.
Imagine if 20+ billion devices are connecting to the cloud (asynchronously) using non secure medium. The connection between gateway is non HTTPS.
An IoT Gateway can be a physical or a virtual device (ex.installed on your computer) which acts as a medium to send data to the cloud. An IoT Gateway does much more than this but that is outside the scope of this blog.
Ingest the Data
In our architecture, we connecting the IoT Gateway to the Cloud (IoT Cloud Core on GCP) via protocols such as HTTPS or MQTT. Devices connect to Cloud through the managed MQTT broker hosted by Google Cloud IoT Core. Cloud IoT Core enforces the devices to connect securely.
Streaming messages are automatically forwarded to Google Cloud Pub/Sub. Cloud Pub/Sub is a shock absorber for both incoming data streams as well as application architecture changes. Even standard devices may have limited ability to store and retry sending telemetry data.
Cloud Pub/Sub provides a globally durable message ingestion service; it scales to handle data spikes that can occur when swarms of devices respond to events in the physical world, and buffers these spikes from applications monitoring the data.
Pipeline
Pipelines manage data after it arrives on Cloud Platform similar to how parts are managed on a factory line. This includes transforming data, aggregating and computing on data, enriching data from other sources, and moving data to its final location. Google Cloud Dataflow is built to perform all of these pipeline tasks on both batch and streaming data.
Storage & Analytics
From Data Flow the data reached the storage. This data can be stored in Big Query which is a data warehouse to perform offline analysis or the data can be stored in Cloud Storage or even more fine tuned using Data Lab and Data Proc.
Analytics is where you extract the information value from the raw or processed data. While you can do some analytics during streaming in the Cloud Dataflow pipeline, much of the analytics processing is over data accumulated in various storage systems.
Often the value of IoT analytics comes from combining data from the physical world with data from other sources, such as customer relationship data, or online information systems. In this solution BigQuery data warehouse is used for long term analytics.
For real time monitoring and alert of streaming data StackDriver Monitoring is used.
Presentation & Application
Last but not the least, we have the presentation or application layer where this data can be used in the Compute engine or App engine or similar Application purposes.
Some of the ways companies are using IoT is Asset Tracking, Smart Utilities, Wearables, Smart City, Smart Homes, Smart Cars, IoT in farming, Health & Fitness and many more.
The 7 Layers of the IoT Architecture
Layer 1: The Things Layer
This layer of IoT comprises of devices, sensors and controllers. Connected devices are what enable the IoT environment. These devices include mobile devices such as smart phones or tablets, micro controller units and single-board computers. The connected devices are the real endpoint for IoT.
Layer 2: Connectivity/Edge Computing Layer
Layer 2 is the connectivity/ edge computing layer, which defines the various communication protocols and networks used for connectivity and edge computing. It is a distributed architecture where IoT data is processed at the edge of the network.
Layer 3: Global Infrastructure Layer
Layer 3 is the global infrastructure layer, which is typically implemented in cloud infrastructure.
Layer 4: Data Ingestion Layer
Layer 4 is the data ingestion layer, which includes bigdata, cleansing, streaming and storage of data.
Layer 5: Data Analysis Layer
Layer 5 is the data analysis layer and relates to data reporting, mining, machine learning etc.
Layer 6: The Application Layer
Layer 6 is the application layer, which comprises of the custom applications that is actually making use of the things data.
Layer 7: People and Process Layer
Layer 7 is the people and process layer. This includes people, businesses, collaboration and decision making based on the information derived from IoT computing.
