Internet of Things as Architecture Pattern

IASA identifies the Internet of Things, or IoT, as a key game-changing technology and an industry trend.  The internet of things is defined as a communication pattern between an embedded compute function – that is, a CPU or other computation device, within another machine or object – which can connect to either other IoT objects and/or a service provider network (or the Internet) for context-aware capabilities.  Typically, the “thing” which has onboard computation capability achieves higher-value capabilities by the interaction with its environment, other “things” in the same domain, or even other computational capabilities on the network.  This trend is evident in many different industrial domains and the enhanced capabilities have widely varying value propositions.  For example:

  • An automobile may have an array of sensors onboard, local and broadband communications, and a telematics control unit, to both coordinate movements to avoid accidents with other cars and provide geolocation data to the driver’s parent, as well as bring services into the vehicle such as maps or entertainment.
  • A wristband with sensors for heartbeat and geolocation may be able to transmit data back to the owner’s running log in the cloud and provide fitness coaching and directions to the jogger
  • A drone with a camera may be able to overfly a farmer’s field and send back imagery which can be combined with weather data to provide optimal watering and fertilizing instructions to the robotic tractor following behind
  • An automated home or office might contain sensors for heating and cooling and communicates with the occupant’s cellphone to warm up the space when returning from a trip away or identify when electricity usage is suboptimal (lights on at night, for instance) to a building manager
  • A chemical manufacturer has an older plant that gets retrofitted with some electronic control systems which can predictively alert headquarters when a high-value pump may be nearing its maintenance or failure window, automatically requesting a technician visit.
  • A parent working with their daughter on a school science project uses some off-the-shelf components to build a system for the science fair to monitor her blood pressure during the day, which a local healthcare provider notices and helps the parent make a new product for the local employers to distribute to their employees to stay healthier.

This trend and type of distributed system creates new business models and benefits consumers with new capabilities for traditional devices, makers, product owners, and system architects.  IoT development can be undertaken by hobbyists through to large-scale product companies in dynamic industries.  The importance of this trend lies in its ability to transform product markets and bring enhanced capabilities – often higher efficiencies, dramatically lower costs or new contextual awareness – to the end owner and operator.

Early Machine-to-machine (M2M) communications consisted of remote machines relaying data to a central computer which acted in a supervisory or controlling role. Decisions were centralized, with unidirectional data flow and security based on isolated network segmentation.  One change which helped start a broader IoT trend has been the gradual adoption of standard internet networking protocols over previous analog or proprietary ones.  In IoT, many systems designers still choose proprietary networking connectivity providers – using open standards protocols on network segments which may offer improved SLA over other options (such as public Internet).  The proprietary or multi-vendor (“open”) network segments are interoperable by the choice of the IP protocols and bridging sometimes does occur.

The convergence of information technology and operational technology (“IT/OT convergence”) was also observed in the early 2000’s as some of these formerly isolated systems could be augmented with communication processes to provide insight into operations and manufacturing processes on a much larger scale, or across geographies.  This provided essentially data streams from distributed sensors to a central computing capability to provide better analysis, like telemetry sent from a rocket back to earth.

Finally, advancements in the robotics fields generated interoperable and significantly smaller (and hardened in some cases) components for the distributed systems so that hobbyists and device manufacturers could explore compute and sensing capabilities in new classes of manufactured products.  Communications modules and sensor modules both became smaller, to the point where it is feasible to put combinations of capabilities – including compute, storage and networking – in form factors as small as a coin.  Typically, these “mini-computers” or embedded computers could now run standard operating systems and programs, and track sensor data from their local environment in real-time.  This data can be processed on the “thing”- or device – or could be shared with proximity devices (peers) or sent to more powerful compute capacity located elsewhere – in a manufacturer’s datacenter or a cloud-hosted environment.  The communications with the device can also be bi-directional, allowing commands, applications, multi-media or other information to be sent to the device as well, enabling new capabilities and new sources of value for commercial systems.

An area of related interest is the nature of these “back-end” or aggregated services provided: with the advanced in machine learning, artificial intelligence and data warehousing, the processing of device data streams from many hundreds of thousands of independent[1] “things” gives unprecedented ability to analyze varied deployments in different conditions and locations to provide insights into how they may be optimized.  This is discussed in the trend on business intelligence elsewhere in the ITaBOK.

[1] Who could ever need more than 100K things?  Brian and Shri, 2016

Internet of Things to the architect

IoT business model changes and use cases

The architect faces both a novel challenge and can add significant value in the design and implementation of IoT devices for their organization.  The IoT trend – though it may be enabled by technology changes- is fundamentally a new business model which enables devices which previously had little or proprietary connectivity outside their “boxes” to add new services which are context-aware for the owner or user of the device.  We have seen some of these above and have a longer list of use cases below: new ideas for these technologies emerge daily!  The architect has two main roles through the design of such a system: 1) to work with the business stakeholders to conceptualize the business model and develop guideposts like cost and schedule for the system implementation, and 2) simultaneously develop the prototype solution in a rapidly changing technical landscape while planning for the future.

From a business model perspective, there is usually a starting “device” which we will modify with hardware, software and connectivity to add capabilities.  The architect will develop the technical specification for these changes and likely contribute to the overall roadmap for this new device, whether that is taking a CF lightbulb and adding programmability to it or a larger set of systems like an automobile or skyscraper and providing incremental value propositions over a longer expected lifetime.

Some of these starting devices may already be in the marketplace or they may be repurposing an internal device or tool into the consumer marketplace.  The architect should consider lifecycle concerns as part of this roadmap: for instance, when will v1.0 ship, how will consumers get the device and get connected, what will support look like, what if a change to the device needs to be made or an upgrade is available (migration between versions)?

One critical business decision to make is how the device will be sold: will it be direct to consumer? Through an established channel? Will it be the same business model we’ve had in the “unconnected” version – a single transaction, or a monthly revenue stream?  Will the consumer aggregate the capabilities themselves – as in a toolkit – or are we providing a true solution out of the box?

While planning the evolution of the connected device, the architect and business manager will likely go through deployment phases such as:

  1. An initial business proposition
  2. An initial (“working”) prototype for internal users or key early customers
  3. A first version in market (possibly a minimum viable product)
  4. Subsequent improvements to either that initial version or fully new purchased IoT devices per roadmap

At each of these phases, the IoT product owner will assess functionality, risk, cost and schedule to the next phase.  The architect can change technologies for each or any component in the system as the prototype evolves to version 1.0.  It is important to ask the stakeholders what their priorities are: whether final system operating costs or speed to market, whether specific functionality or lower risk profile.

The diagram below gives us a first look at this distributed, “Internet of things” systems architecture and the components we will talk about in later sections.

The device, or “thing,” which will be connected to its environment is represented by the blue gears on a blue background.  It would typically contain a CPU – often described in domain terminology as an MCU, TCU, GPU, etc. – with onboard storage, networking components, power, and sensors.  This is usually fabricated on a “green board” or prototype board or normal motherboard backplane (at scale), and these components will live in a single assembly (mounted in the original device).  The communications components are often prefabricated microchips with communications antennas built in for protocols such as Zigbee, X10, TCP/IP (WiFi or even self-routing), or Bluetooth.  The sensors similarly are “off-the-shelf” and may be cameras, infrared, temperature, motion, location/position (including GPS), pressure/impact, voice or sound-activated or recording, equipment monitoring[1], etc.  These are a critical part of the business case and may have industry standards which apply to their usage.

The device often has an interface for the owner, operator, or user to interact with directly, either on the device itself (such as a touchscreen) or able to communicate remotely to the device, such as a Google Nest™ unit which interacts with the home HVAC system located elsewhere.

[1] Such as for a pump

Image courtesy of Brian Loomis, BVStrategic.com, 2016

The device may interact with the local environment (shown as a cloud, representing weather sensed by the device) and may have local-area communications with other similar devices.  If the device does not have a direct human interface, we call in “headless” indicating that the device is controlled remotely, for instance on an underwater robot.

Finally, IoT devices usually communicate over wide-area networks to a datacenter for additional processing.  This wide area network could be a private network or over internet, possibly even a connection supplied by the owner of the device (e.g., home WiFi connection or a tethered phone IP connection).  The backend datacenter could be a public cloud service or private datacenter hosting as well, and information services provided there can be either sent back to the device or may be accessible through other means (see discussion below).  The diagram is representative only.  Many IoT systems do not have all of these components.

Use cases for IoT

There are a myriad of different use cases so far developed for Internet of Things and this section only provides a short list of some possible areas.  IoT applies in almost every industry and each domain has many current exploratory use cases under active prototyping.

Some example industries, with associated use cases, include:

  • Agriculture – automated tractors, surveillance drones
  • Automotive – V2V communications (accident avoidance), self-driving cars, infotainment, performance
  • Construction – smart buildings
  • Fashion – color-changing fabrics
  • Health – air & water purity, wearables for exercise, tracking metabolic processes (insulin)
  • Industrial (process and discrete manufacturing) – predictive maintenance, robotics for hazard remediation / environmental monitoring
  • Military – drone swarms, surveillance, force protection
  • Realty and property management – single family home sales, big data market assessments, low-income housing
  • Retail sales – proximity detection and advertising
  • Rocketry – video feeds, weather

The following image represents the companies involved in just the “wearable” market as of a couple of years ago.

Image courtesy of Craig Dudenhoofer, http://wearableworld.com, 2012

IoT technical platform changes

This section describes some of the changes which are important for the architect to understand on the technical side of the design of IoT systems.  Many of these areas have changed significantly over the last ten years or are new areas unfamiliar to architects who have worked primarily on software-only projects.

The IoT system (as shown above) is a composed system of both hardware, software and networking sub-systems.  The hardware components of the actual device are small board-sized elements and require electrical engineering as well as embedded systems skills.  Some typical backgrounds for architects in this area would include experience with Windows Embedded devices, small AMD or Intel boards (e.g., Galileo), or maker experience with hobbyist boards like Arduino or Raspberry Pi.

The networking experience required ranges quite broadly depending on the use case: if the operator of the device is using wireless home or business-provided internet routing, then traditional TCP/IP methods are all that is required.  If the device has peer-to-peer capabilities, then familiarity with the selected standard is useful.  If the device has WAN communication capability – as found in automotive or robotics applications – then the architect will need to understand private networking as provided by AT&T, Orange, Verizon, Deutsche Telekom, Cisco, etc.

The architect will also need to select much of the software platform for the datacenter, including ingress or ingestion of telemetry from the large number of devices, parsing and routing of this telemetry to command and control systems and back to devices, interaction with user and datacenter (possibly by mobile phone), and any higher level analytical capabilities.

Many of these skills may be new to the architect, and by virtue of selecting many of these technologies, the architect may play a role in training the broader team on these.

Architecturally significant requirements

As the business case for IoT is often different from previous product implementations, the architect should evaluate the quality attributes for the ones which are most important in this specific development.  Some of the attributes which can be significant for IoT projects include:

Cost: the architect may have to build a new cost model for the IoT device, including both initial deployment costs (which may include previously inconsequential costs like new product marketing, logistics, etc.), and operating costs.  The parameterization of this cost model should be revisited frequently as the IoT application goes from prototype to selection of version 1.0 vendors and more is known about the adoption curve.

Security: since the device is connected either to other devices, the Internet or back-end datacenter assets, and possibly contains PII, a full security risk profile and remediation effort should be undertaken.  The device could be hijacked, leading to PII loss, corporate data loss, or potentially modifying the object’s behavior in illegal ways.

Scalability: One of the goals for most IoT projects is the large number of connected devices, often numbering in the millions.  The architect plays a critical role in identifying and correcting scale issues as the system is developed; these could arise in any implementation area from the on-device performance, to the network traffic to the ingest and message routing in the datacenter.

Performance: Often being a consumer-facing device, performance can be measured with HMI metrics.  There may be other, domain-specific metrics required such as safety-critical response timing or verification of command transmission.

Reliability: If the product is a new offering, the usability and reliability of the device for purpose affects market perception and ultimately sales of the device.

Usability: Also, if the product is new in market, the purchaser or owner of the device should be able to quickly determine that it is working initially.  Prior internal product offerings may not have been subject to rigorous user-acceptance criteria and some “productization” may be required in this area.  For example, if the onboarding of the device requires network access, the device may have to alert the user to enter a passcode, which – if a complex process – could lead to dissatisfaction.

Manageability: Speaking to later in the lifecycle, there will ultimately need to be at least software changes to the device either to fix known issues or to add new capabilities.  The architect should be able to sketch out how these updates will happen.

Technical areas of architecture

This section attempts to describe the minimum architecture and its artifacts for a “standard” IoT program.  The Internet-of-Things starts with the design of the “thing” at the heart of the diagram, itself a system.  The “thing” or device hereafter will have interfaces to a number of other systems, including a datacenter and optionally other things (local mesh), or owner HMI.[1]  The software systems (generally in the device, the datacenter, and optionally mobile platforms) will have software architecture artifacts.  The device will have a hardware specification.  The networking segment(s) may have a specification.  Each interface will have a interface architecture comprised of message formats, sequence diagrams, translations and units-of-measure, and error handling.  It is important that system quality attributes are traceable to subsystems from the beginning, so that overall system quality attributes and service level can be determined through regular scale and other tests.  Your IoT project may require significantly more technical architecture design, but this section should cover a scalable set of deliverables and considerations to reach that goal.

The technical architecture will also include views on the consumer aspects of the project:

  • Legal use and sharing of data
  • Sales channel integration
  • Production capacity (especially hardware)
  • Pricing (COGS)
  • Security standards and evaluation by third-parties/regulators

The initial set of architecture artifacts comprises:

  1. Overall system conceptual architecture
  2. Device architecture
    1. Hardware selection, prototyping, final design
    2. Software platform selection, including message passing infrastructure components
    3. Custom software application architecture
  3. Network architecture
    1. May include provisioning, certificate issuance/identity verification of devices, local contracts and availability
  4. Datacenter software architecture (pilot, “scale unit” and production sizings)
    1. Device I/O, telemetry and commands (via network)
    2. Administrative / management HMI
    3. Data storage (including time frame for retention)
    4. Billing and provisioning interface
    5. Partner or internal data aggregation facility (reporting, optional)
    6. Advanced software architectural components (“smart” services, optional)
  5. Owner/operator HMI
    1. Web portal for owner
    2. Mobile application (optional)
  6. Peer device interfaces and subsystems (optional)
    1. May include local networking capabilities (non-WiFi or non-WAN)

The following three sections describe some of the process for the hardware device, the communications and networking, and the datacenter services for IoT projects.

[1] HMI, or Human-machine interface, is a generic term for how the owner/operator may interact with the device.  This may be a touchscreen on the device itself or it may be a remote application such as from mobile phone.

Embedded technology – the device

The device itself has a complete stack of capabilities, from hardware, through software platform to custom applications which are deployed onto the “thing.”  Often, separate hardware and software teams must coordinate their activities in a series of integrated builds.  From the hardware perspective, the builds may start as prototype boards (see Raspberry Pi image below), and then move to either a custom PCB fabricator/assembler or a completely outsourced hardware model.  The hardware will have a software kernel running onboard, for deployment of applications, including the client side of the message passing stack.  This kernel could be a  primitive OS such as Arduino, or a more featured LINUX kernel (such as Blackberry’s automotive QNX kernel) or Windows Embedded, or even a more full-featured OS which has a full TCP/IP stack, a standard development platform and web server,  and multithreading capabilities.

Image courtesy of Raspberry Pi 2 team, https://www.raspberrypi.org/products/raspberry-pi-2-model-b/, 2016

Communication protocols and the network

The device communicates with other devices, owner/operator HMI and back-end datacenter services over one or more network subsystems.  Each communication endpoint may require a different network but often there is incentive to reduce cost and consolidate to one or two networks for this communication; wireless already in place with the owner/operator of the device is preferred over for-fee private telco networks or public phone connectivity (tethered).

Sample RabbitMQ topology, image courtesy of http://image.slidesharecdn.com/consumingrabbitmq-150219062454-conversion-gate01/95/consuming-rabbitmq-at-ttl-10-638.jpg?cb=1424851986

Once the communication network is selected, the protocol for message passing should also be selected.  This may be a domain protocol (such as NGTP or AMQP in telematics) or may be a generic message passing protocol such as MQTT or simple MQ.  The domain protocols offer two advantages: the message passing sequences are pre-defined (and testable) for domain operations, and they are often built into higher-order software components for easy implementation (for example, Apache Kafka and Azure IoT Hub support MQTT with only minor configuration required).  A sample AMQP frame is shown below, indicating the wire packaging required for these byte-ordered protocols:

Sample AMQP frame, image courtesy of https://paolopatierno.wordpress.com/2015/07/23/amqp-on-the-wire-messages-content-framing

Peer-to-peer (device-to-other device) or mobile phone to device patterns may use custom protocols or mirror the datacenter communications protocols for convenience.  The architect will be responsible for selecting these communication protocols, possibly from RESTful API/HTTP, AMQP, CoAP, OMA-DM, or MQTT, or as built in to the software stack,such as Azure, .NET Framework, IBM MessageSight, Apache Kafka, RabbitMQ, or others.

Sample AMQP workflow/interaction diagram, courtesy http://www.openamq.org/tutorial:soa

The protocol layer is important since it is often implemented as a whole subsystem and quite likely bought rather than custom developed.  This provides higher reliability with (ideally) reduced testing effort.  For example, a single library may be able to implement on both the device and the datacenter message passing using MQTT.  On the datacenter, this will provide operations to perform:

  1. Decryption of message
  2. Identification of sender and message type (by certificate)
  3. Unpacking or translation of message body
  4. Routing and storage
  5. Hand-off to processing elements (also known as message ingestion)

Note that testing is further reduced by selecting matching pairs of versions of the protocol on both the device and the datacenter stack.

Internet-available services from the datacenter

As the datacenter aggregates messages to and from the array of devices fielded, it has itself many possible services to offer:

  • On the device-facing side, this is generally a telemetry and command/control operations interface through a device gateway or scalable aggregation mechanism
  • To the consumer, the datacenter can provide mobile access to the telemetry and C&C functions directly or through a device owner portal (think “Manage My Device.com”). This may involve other authentication sources from what the device uses and may involve push notifications as well.
  • To the device manufacturer (or partners), CRM functionality, usage functionality (like NewRelic), and advanced services are provided by the datacenter. Often the advanced services involve the “smart” part of the device – it’s adaptability to function and recommendations to tune its usage: this is often done through data analytics (HD Insight/Hadoop, Machine Learning, Azure Search, Elasticsearch, etc.).
  • To the sales channel, a means of provisioning, deprovisioning and upgrading devices already deployed as well as tacking sales contacts (CRM-like functions).

Each of these is typically determined through the business plan and varies significantly between IoT implementations.

Sample Azure IoT services, courtesy Microsoft.com 2016

Perspectives

Business Technology Strategy

BTS is probably the most important aspect for decision makers to work towards clarity on during the course of any IoT implementation, since it combines information from simultaneously evolving business models and technical platforms.  Both the business model and the underlying technology enablers (and costs) will change radically during the course of a project and changes occurring on one side also impact the other.  Similar to other aspects of the system design, the business technology strategy incorporates change and also has specific business models which should be visible to the team for directional guidance.  Specifically, aids like the business model canvas and SWOT analysis – though confidential from competitors – should summarize the current business thinking well enough for implementation.  Regular synchronization points should be planned for after each iteration, so that any changes to the business model are conveyed to implementation teams.

Minimum Viable Product and other approaches may be appropriate for an IoT project.

Design

As noted in the technical description above, the generic IoT design consists of multiple sub-systems, requiring versioned designs themselves.  Often a tool like swagger.io or even well-documented WSDL can provide versioning for the multiple interfaces required in the system.  There will also be software and hardware architecture design documents and drawings which will be versioned.  Since message passing is a fundamental construct, some auxiliary diagrams or tables may be required for translations of messages.

Since the designs are versioned, we expect that the implementation artifacts will be as well, so multiple deployments should be planned for in the schedule.  All versioned designs should be kept in a repository where they are visible to all partners.  Automated deployments should be pursued for IoT projects, whether following a specific DevOps approach or simpler tooling.

Quality Attributes

These are described in detail in the section above on ASR’s, and IoT projects generally need to well-define and manage the following quality attributes:

  1. Cost
  2. Security
  3. Scalability
  4. Performance
  5. Reliability
  6. Usability
  7. Maintainability

These tend to be inter-dependent and dynamic so the approach should define the service level for each (in quantitative estimated terms like $$$ for cost, or mixed terms as in system performance), for the initial prototype version, the “model” or “stamp” or “scale unit” sized deployment, and the full-up production deployment footprint.

Since early information may be incomplete, multiple estimates and verification (through testing and/or market research) should be performed on a regular basis.

IT Environment

The IoT environment is very complex and it might even be questioned as to whether the “IT” environment is the right home for this set of discussions.  Should IT or even a product team with technical implementation skill be the focal point of the multi-vendor coordination role that is required for an IoT project?  Fundamentally this is a system-of-systems integration role among multiple coordinated implementation teams with very different deliverables.  There will be a set of tools, partner(s) and technologies for the network segment, for the IoT hardware itself, and probably multiple software implementers.  The key for this role is to constantly know where the overall system state is… that is, where each sub-system is accurately at, and what changes to scope are required, and provide a forum for resolving conflicts and issues that arise.

Human Dynamics

Human dynamics is probably the least changed through an IoT project.  Two specific areas should be addressed: discussions between coordinated partners on the implementation teams, and how to bring the voice of the customer into the effort.  This latter one may take the form of an advisory board, early adopter customers or the like, but pay attention to how much the advisor knows (some details may remain confidential until after launch), and the advisors may not understand the impact of changes to technical platform, or scope of changes they could suggest.  This lends itself to a guided advisory board model.

Maturity and capabilities

The following table illustrates some potential ways to assess a team’s or an individual architect’s maturity and skills for an IoT project.  It is a draft only and meant as a guide to developing your own maturity through the CITA proficiency levels.

   BASIC MODERATE ADVANCED
Architecture process Embedded System Architecture

Knowledge of different connectivity technologies

Knowledge of 3GPP/LTE

Knowledge of Cloud and Analytics

Knowledge of IoT protocols (MQTT, AMQP, etc)

Knowledge of Battery and power technology

Utilize Embedded system knowledge and connectivity technologies to create a node

Utilize knowledge of cloud and analytics technologies to receive data from nodes and extract relevant content

Know how to provision devices from cloud

Knowledge of Billing strategies

Able to deploy networks for specific use cases

Portfolio-based approach

Architect personal skills Able to talk about connectivity in terms of bandwidth vs. power requirements

Able to decide which IoT protocol to choose for communication between the gateway and the nodes, and the gateway and the infrastructure

Able to decide how to analyse the data received

Often the do-er and trainer of others (Stratus team)

Seeks out individual projects

Able to create an end-to-end architecture for a given use case Able to decide billing strategies

Able to create architectures which scale to diverse use cases

For further review (Resources)

This section includes references for IoT in several different categories:

Business value

Technical (maker or major technology vendor)

Architecture techniques

Authors

TBDThis perspective on the trend towards IoT systems was written by Shrikumar Sharma, who is a nice gentleman.  Quae cum ita sint, Catilina, perge, quo coepisti, egredere aliquando ex urbe; patent portae; proficiscere. Nimium diu te imperatorem tua illa Manliana castra desiderant. Educ tecum etiam omnes tuos, si minus, quam plurimos; purga urbem. Magno me metu liberabis, dum modo inter me atque te murus intersit. Nobiscum versari iam diutius non potes; non feram, non patiar, non sinam

brian loomisBrian Loomis also wrote this perspective: he is both a CITA-Professional and trainer for IASA Global.  His “day job” is as an enterprise architect for large organizations and leading software teams to deliver products in education, healthcare and manufacturing.  He has presented at industry and academic conferences including ACM SuperComputing, International Telemetry Conference, and at national business IT conferences.  Brian served as an officer in the United States Air Force and holds a Master’s degree in Computer Science from California State University and a Bachelor’s degree in Computer Engineering from Princeton University.  Check out his profile on the IASA instructors web site, his company (http://www.bvstrategic.com) or drop him a line at bwloomis404@gmail.com if you have an architecture question.