Scalable Architecture for the Internet of Things An Introduction to Data-Driven Computing Platforms Ervin Varga, Draško Drašković, & Dejan Mijic Scalable Architecture for the Internet of Things An Introduction to Data-Driven Computing Platforms Ervin Varga, Draško Drašković, and Dejan Mijić Beijing Boston Farnham Sebastopol Tokyo Scalable Architecture for the Internet of Things by Ervin Varga, Dejan Mijić, and Draško Drašković Copyright © 2018 O’Reilly Media, Inc All rights reserved Printed in the United States of America Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472 O’Reilly books may be purchased for educational, business, or sales promotional use Online editions are also available for most titles (http://oreilly.com/safari) For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com Editor: Brian Foster Production Editor: Melanie Yarbrough Copyeditor: Jasmine Kwityn Proofreader: Charles Roumeliotis February 2018: Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest First Edition Revision History for the First Edition 2018-02-13: First Release The O’Reilly logo is a registered trademark of O’Reilly Media, Inc Scalable Architec‐ ture for the Internet of Things, the cover image, and related trade dress are trade‐ marks of O’Reilly Media, Inc While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work Use of the information and instructions contained in this work is at your own risk If any code samples or other technology this work contains or describes is sub‐ ject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights 978-1-492-02412-5 [LSI] Table of Contents Preface v Internet of Things (IoT) Introduction to IoT The Architecture of a Data-Driven Solution Evaluation Criteria for IoT Platforms Active Load Control Case Study Summary 12 14 22 Amazon Web Services (AWS) IoT 25 Introduction to AWS IoT Overview of the Architecture Setting Up the Ecosystem Event Management and System Dashboards Application Integration Layer Security Building an End-to-End Example Summary 25 27 31 39 40 41 41 42 Microsoft Azure IoT Suite 45 Introduction to Microsoft Azure IoT Suite Overview of the Architecture Setting Up the Ecosystem Event Management and System Dashboards Application Integration Layer Security Building an End-to-End Example 45 46 52 58 63 70 73 iii Summary 74 Mainflux 77 Introduction to Mainflux Overview of the Architecture Event Management and System Dashboards Security Building an End-to-End Example Summary 77 78 83 86 88 94 EdgeX Foundry 97 Introduction to EdgeX Foundry Overview of the Architecture Setting Up the Ecosystem Event Management and System Dashboards Application Integration Layer Security Building an End-to-End Example Summary 97 99 102 104 107 109 110 112 A Conclusion 117 iv | Table of Contents Preface The Internet of Things (IoT) is heralded as the fourth industrial internet, and participate in machine-to-machine and machine-toperson use cases on a massive scale The common trait of all these IoT use cases is to efficiently handle the following essential jobs: connecting and managing billions of devices, transferring data over the network, storing data, and processing data Apparently, data has a central place, so we may freely announce that IoT is data-driven Computing associated with data is there to squeeze out knowledge, and provide the ability to automate many aspects of our environ‐ ment The objective is to deliver new value-added business services that were not possible before Any cutting-edge technology/paradigm, like IoT, is surrounded by a vivid, dynamic, and growing community The participants seek to gain knowledge and experience in the novel domain The interrela‐ ted environment is permeated with various overlapping technologi‐ cal alternatives that are frequently accompanied by hype It isn’t surprising then to encounter terms like Massive IoT, Industrial IoT, Critical IoT, Web of Things (WoT), and Internet of Everything (IoE) Furthermore, we encounter stuff like digitization on one hand, and Invisible Computing, Transparent Computing, Edge Computing, and Fog Computing on the other (these are some of the most popular phrases) Our aim is to find a common denominator among these elements, rather than delve into a convoluted elabora‐ tion of how to properly classify them For example, in terms of device connectivity we may classify lowpower network technologies into the following broad categories: unlicensed spectrum for short-range and mixed-performance com‐ v munication (WiFi, Bluetooth, Zigbee, etc.), unlicensed spectrum for mixed-range and low-performance communication (SIGFOX, LoRa, etc.), and cellular for mixed-range and mixed-performance communication (EC-GSM, LTE-M, NB-IoT, etc.) Regardless of which category they belong to, all of these technologies seek to bal‐ ance dependability, performance (throughput and latency), and cost/complexity If you want to learn more about how cellular net‐ works are adapted to the needs of IoT, read the Ericsson whitepaper Cellular Networks for Massive IoT A principal enabler of massive adoption of IoT use cases is the exis‐ tence of an efficient IoT platform It combines device connection/ management and service enablement functions It may be treated as a key building block in IoT solutions Custom applications are craf‐ ted on top of an IoT platform that handles all the mundane work regarding devices as well as data analytic capabilities An IoT plat‐ form is like middleware for distributed applications IoT platforms are the topic of this report, and are superb examples of scalable architectures for IoT For a good commentary about the importance of IoT platforms and data management in IoT, refer to The Platform Transformation—How IoT Will Change IT, and When by Matthew J Perry (O’Reilly) It is impossible to provide exhaustive coverage of IoT platforms in this short report Instead, we have chosen to focus on the following goals: • Teach you how to become proficient with some concrete IoT platforms • Help you sharpen your knowledge by suggesting many refer‐ ences for further study • Offer different perspectives on IoT topics • Provide some comparative analysis between various industrial IoT platforms vi | Preface This work is structured as a report rather than a fullblown book Consequently, its content is presented in a condensed form with many references for further reading We have tried to provide enough content for you to understand the material even without consult‐ ing outside resources Nonetheless, to gain deeper knowledge you will need to review the various books, webinars, and blogs we highlight throughout this report Contents of This Book This report is comprised of five chapters: • Chapter is a general introduction into the world of IoT • Chapter presents Amazon’s AWS IoT platform • Chapter presents Microsoft’s Azure IoT platform • Chapter presents the Mainflux IoT platform • Chapter presents the EdgeX Foundry edge component Chapters Chapter through Chapter follow a similar outline, and a table is provided at the end of each of those chapters to summarize the IoT platform discussed therein (using the set of evaluation crite‐ ria set forth in Chapter 1) There is also a Conclusion that wraps up this report Preface | vii (i.e., IoT middleware) Therefore, there are currently no UI dash‐ boards in the EdgeX system A good candidate for a UI solution would potentially be the Client Registration service, where it would be useful to list and administer data and user credentials needed by the Export Distribution service to push the data into the cloud Regarding the event flow and management, the EdgeX system pro‐ vides a lot of facilities to control and configure the way data will enter and traverse the EdgeX stack Data flow starts from devices, which either push the data asynchro‐ nously, or are being polled for measurement updates (Figure 5-5) A set of Data Services form the various adapters, capable of accepting data via different industrial protocols (for example, Modbus or MQTT) Data is then pushed toward Core Services, which put the data in the internal data format and store it in the database (Figure 5-6) Event Management and System Dashboards | 105 Figure 5-5 The UML sequence diagram of a typical data flow 106 | Chapter 5: EdgeX Foundry Figure 5-6 The core data entity-relationship model Core Services then push the data further up the stack, via Export Services, where the data is additionally processed and filtered Finally, the Export Distribution service pushes the data toward the remote cloud, using the necessary credentials and configuration pre‐ viously registered in the export database via the Export Client Application Integration Layer All EdgeX services offer REST APIs that applications can use How‐ ever, by the nature of the systems on which EdgeX software will run, most of these APIs should be used sparingly—especially when we talk about Core Services These are mostly system configuration API endpoints, used by the various system-software components run‐ ning on the gateway itself Vertical IoT applications however will be built on the top of the IoT cloud to which data will be forwarded via the EdgeX IoT gateway Otherwise, low-latency vertical applications will run on the gateway itself But in both cases, these applications will be connected to EdgeX system via a northbound interface pro‐ vided by EdgeX Export Services, and obtain the data that Export Distribution will forward to them after some internal processing Application Integration Layer | 107 The topmost layer of the EdgeX stack, called Export Services, con‐ tains primarily two services: Client Registration and Export Distri‐ bution (Figure 5-7) While Client Registration provides REST APIs to store the configuration and credentials needed to access remote cloud or on-gateway applications, Export Distribution uses these credentials to appropriately pump out the data and feed it to appli‐ cations Figure 5-7 The structure of the Export Services layer, and its interrela‐ tionship with the environment Data flow from device toward the application on the northbound interface can be described via an UML sequence diagram, as depic‐ ted in Figure 5-8 108 | Chapter 5: EdgeX Foundry Figure 5-8 Data flow from device toward the application on the north‐ bound interface Most often that the gateway is sending the data toward the remote cloud middleware, thus it acts as an HTTP or MQTT client How‐ ever, an on-gateway application would most often be configured to accept ZMQ notifications, if it is not capable of providing HTTP API endpoint, or there is no MQTT broker on the gateway Security There is an ongoing effort by the EdgeX Security Working Group to design and improve the security of the IoT gateway The following recommendations are currently in the backlog: • Handling of secret keys • Certificates • Data encryption Planned and ongoing activities are listed in the following table: Security | 109 Table 5-1 Table caption here Keys Create, Read, Update (attributes), Delete CA Data protection Simple out-of-the-box self-signed CA Encrypt/decrypt Store (external keys) and support policybased key export Support external CAs Sign/verify Certificate status queries (OCSP) MAC Building an End-to-End Example For providing more insights into EdgeX internals, we will use an example of industrial communication via WiFi, sending the analog measurements from robotic arm In order to simulate the equip‐ ment, we will use a WeIO board, an inexpensive prototyping board programmable in Python (Figure 5-9) The board can be connected to the EdgeX gateway via wired or wireless connections For simplic‐ ity, we will use a 2.4GHz WiFi connection 110 | Chapter 5: EdgeX Foundry Figure 5-9 The prototype board for the example As shown in Figure 5-9, the board has several analog GPIOs to which a measured point can be connected We will use a potentiom‐ eter connected to these pins in order to simulate variable analog input The following simple Python program is used to take a sample potentiometer analog value every second and publish it over MQTT: from weioLib.weio import * import paho.mqtt.client as mqtt def setup(): attach.process(edgex) Building an End-to-End Example | 111 def edgex(): # The callback for when the client receives a CONNACK # response from the server def on_connect(client, userdata, flags, rc): print("Connected with result code "+str(rc)) # The callback for when a PUBLISH message is received # from the server def on_message(client, userdata, msg): print(msg.topic+" "+str(msg.payload)) def on_publish(client, userdata, mid): print("mid: "+str(mid)) client = mqtt.Client() client.on_connect = on_connect client.on_message = on_message client.on_publish = on_publish client.connect("localhost", 1883, 60) # Start looping client.loop_start() while True: val = analogRead(31) # read pin 31 client.publish('/edgex/value', val) delay(1000) Summary EdgeX Foundry is a new industrial initiative that shows great poten‐ tial Unifying and standardizing computing processes on the edge is a complicated task, as IoT in general lacks standardized procedures An IoT gateway makes up part of practically every industrial setup Building and deploying software for those kinds of gateways cur‐ rently consists of many repeating tasks by various companies (Figure 5-10) Creating a single generic software base would help avoid these repeating procedures, and provide the best practices already implemented and ready for deployment 112 | Chapter 5: EdgeX Foundry Figure 5-10 Tiered IoT deployment scheme Unification of processes is very important for security as well Secu‐ rity experts from various companies in the EdgeX consortium formed a specialized Security Group to treat this subject and find the best approaches to protect the gateway data, mitigate security risks, and prevent attacks Industrial applications also demand various certifications, project longevity (long-term support), and various proofs that the code was properly tested under various conditions It is often very difficult for a single company to provide these kinds of features, but it is possible for a larger enterprise consortium, and this is exactly what the EdgeX Foundry project is trying to achieve The first implementation of the EdgeX system was written in Java, but the newly optimized implementation in Golang is advancing very quickly This implementation exhibits a smaller memory foot‐ print—something that is very important for more constrained industrial gateways, usually based on the ARM architecture It also shows faster bootup times and more robust deployments, because Go services can be compiled as a single static binary and deployed in so-called “FROM SCRATCH” Docker containers that enable a size of container image of just a few megabytes Summary | 113 Table 5-2 Table caption needed Trait name Device bindings Trait description Many industrial protocols Additions possible via custom device services Analytics Enabled via Export Distribution service Additionally, provided via an external cloud application or local application that gets notifications or does local REST calls Visualization No dashboards Can be added via external cloud applications Rules engine and alarming Integral part of Support Services layer Security Integral part of the system, via dedicated Security layer License Apache-2.0 Deployment technology Docker containers Auto-scaling Can be scaled out via API gateway proxy (currently NGINX), but usually run on edge gateway, without special need for scaling (however, need for HA persists) Device data persistence MongoDB Also, filtered data is exported to external cloud storage Management database MongoDB Implementation language Java and Go Data model JSON Finally, as with any open source project, EdgeX’s success also depends on the community EdgeX is very well positioned in this regard, as it is managed and orchestrated by the Linux Foundation, an umbrella organization that puts a lot of effort in applying best practices from the open source world on every aspect of the project: • EdgeX Foundry code is Apache-2.0 licensed • Clear copyright information (via SPDX headers) • DCO (Developer Certificate of Origin) and other legal checks are put in place • Dedicated GitHub organization and dedicated DockerHub space • Community mailing lists and chat provided (via Rocket Chat) • Project wiki pages are informative and well maintained • Project website lists a lot of information and is regularly upda‐ ted The Linux Foundation also provides testing facilities, Jenkins servers, and CI infrastructure 114 | Chapter 5: EdgeX Foundry As it has been well received by both the community and the indus‐ try, EdgeX Foundry has a good chance for success Summary | 115 APPENDIX A Conclusion Our aim with this report was to help you better understand IoT, teach you how to work with some concrete IoT platforms, provide different perspectives on the topic of IoT, and provide comparative analysis between various industrial IoT platforms In Chapter 1, we gave you a theoretical framework to reason about concepts such as data-driven computing, IoT, edge computing, IoT platforms, key quality attributes associated with scalable architectures for IoT, and evaluation criteria for IoT platforms We also provided a case study to showcase the transformative power of IoT Subsequent chapters elaborated about concrete IoT platforms as well as an edge compo‐ nent to make the text of Chapter more comprehensible Our selection of IoT platforms was just one possible sample We have presented two major commercial cloud IoT offerings (AWS IoT and Azure IoT), one open source unified IoT platform (Main‐ flux), and a special edge component (EdgeX Foundry) Both cloudcentric products contain their own edge component, although they could work with EdgeX Foundry, too Mainflux may be executed everywhere There are many other IoT platforms (both open source and commercial) beyond the ones covered in this report Here’s a nonexhaustive list: • Eclipse Kura • Kaa IoT • Thingsboard • PTC ThingWorx 117 • GE Predix • Google Core IoT Enterprises will need to judge their use cases to select the proper IoT platform when implementing IoT-based solutions There is no onesize-fits-all approach For example, cloud-centric systems aren’t going to perform well for use cases including autonomous vehicles or drones GE Predix and EdgeX Foundry are optimized for indus‐ trial IoT Moreover, many platforms are implemented in Java, while Mainflux utilizes Go The implementation language could also be important, if an organization would like to contribute specific exten‐ sions to the platform At the end of your journey through this report, we hope that you have gained a good understanding of IoT as well as the importance of IoT platforms as key building blocks in IoT solutions Moreover, we hope that you gained some intuition of how concrete IoT plat‐ forms work We also hope that you will explore the materials we have suggested for further reading, which we tried to put into con‐ text so you can choose which ones are most suitable to your inter‐ ests and enterprise needs 118 | Appendix A: Conclusion About the Authors Dr Ervin Varga has been in the software industry as a professional software engineer since 1994 and today is the owner of the software consulting company Expro I.T Consulting and a technical advisor of Mainflux In addition, he is an IEEE Software Engineering Certi‐ fied Instructor, and an associate professor at the University of Novi Sad, Serbia Ervin is also author of several books and papers for sci‐ entific journals and conferences and is an author of the popular evarga/jenkins-slave Docker image Dejan Mijić is a software engineer interested in design and develop‐ ment of distributed systems He is the principal contributor of Mainflux, an open-source, scalable and versatile IoT platform He holds an M.Sc in Computer Science from University of Novi Sad Draško Drašković is an IoT expert with over 15 years of professio‐ nal experience He hacked embedded Linux SW and HW device drivers, designing complex wireless systems in telecom industry Drasko earned his reputation in open-source community as an author of numerous projects - like WeIO or Mainflux He is one of the main contributors of the Linux Foundation’s EdgeX Foundry project Drasko is a vivid conference speaker, sharing his thoughts on modern software design and how Internet should be decentral‐ ized He holds a MSc in Electrical Engineering from Belgrade Uni‐ versity ... (O’Reilly) The Architecture of a Data-Driven Solution The architecture plays a central role in any data-driven software sys‐ tem The architecture bundles together pertinent quality attributes of the. .. Edition Revision History for the First Edition 2018-02-13: First Release The O’Reilly logo is a registered trademark of O’Reilly Media, Inc Scalable Architec‐ ture for the Internet of Things, the. .. overview of the Internet of Things (IoT) We will cover the data-driven computing paradigm and asso‐ ciated architectures, desired quality attributes of an IoT platform, and evaluation criteria for