DevOps from a process architecture perspective
Market pressure demands that organizations continuously respond to ongoing changes and evolving environmental factors. The interplay between different market forces requires enterprises to react and adapt to change more quickly than ever before. This gave rise to what we call ‘chaordic organizations’ and created a stronger reliance on software for the development and delivery of appropriate products and services. Thus, software processes are becoming an integral part of enterprise processes. Current methods of modelling software process reconfigurations are limited in their ability to consider multiple enterprise perspectives and inform selections from alternate configurations. For the rapid and frequent delivery of new software product features and service innovation, the DevOps approach overcomes some of these challenges by enabling;
- The promotion of a culture of collaboration and information sharing between multiple teams—the traditional approach of having organization silos with defined boundaries and handover points is discouraged, and team members are expected to collectively collaborate towards the attainment of enterprise objectives.
- The automation of activities in the overall software development process, through the introduction of software tools and custom development of scripts, thus shortening the time required for new feature development and bug fixes through reduction of manual effort—this enables software teams to deliver more frequent releases to customers and the user base.
- The usage of feedback loops to continuously improve software development processes and the development of product features, through the monitoring and measurement of various software process and technical metrics—these metrics are then interpreted and utilized for overall process improvement.
The framework and the enablers align as shown figure below: A culture of collaboration and information sharing between multiple teams is depicted by Agile disciplines. Automating activities in the overall software development process is depicted by Continuous Integration. Using feedback loops for continuously improving software development processes is depicted by Continuous Process Improvement. .
Figure 1 DevOps framework – Based on work performed by Intel
The framework of figure 1 depicts the DevOps process design, systems and tools development and deployment, and social and organizational issues. The outcomes of DevOps are Continuous Integration (CI) and Continuous Deployment (CD) of product functionality and infrastructure setup. These characteristics are not unique to DevOps, and indeed, are generally applicable to enterprises with respect to enterprise agility and enterprise digital transformation.
DevOps from a practice perspective
CI and CD fit into the overall DevOps process. The simplified DevOps practice consists of Production Management, Development and Automation Engineering streams. Development consist of DevOps Engineering, QA Engineering and Software Engineering. At the centre of DevOps is the development stream. This stream is depicted in yellow in the framework and labelled ‘Agile’. The Automation Engineering is labelled Continuous Integration (inclusive of Continuous Delivery). The process is underpinned by the Continuous Process Improvement stream.
Quality Assurance Testing
Once a feature has been developed, it should be tested (functionally and/or end-user testing) before it goes through the Continuous Delivery process. This testing can be carried out by Quality Assurance engineers in at least two ways:
- they can retrieve the committed code from the code repository and test it in a controlled environment
- they can collaborate with the software engineer to quickly validate the functionality before the codebase is committed to the code repository.
Consideration: The degree to which Trunk Based Development is implemented, serves as a best practice guide. Trunk Based Development (TBD) is a development process in which all developers of a deployable unit commit to one shared branch under source-control.
The enterprise is assumed to have both periodic and fixed release cycles of appropriate duration. A release planning activity is carried out at release initiation, which results in a release backlog; this artefact is then used to plan out individual sprint iterations. Two of the possible alternatives are;
- The release backlog is produced once and remains static throughout the release duration
- The release backlog is revisited at the beginning of every sprint and “groomed” (i.e. reordered and re-estimated), based on on-going change in circumstances and priorities.
Consideration: Agile implementation in the organization
To reduce product delivery duration, some product testing can be automated by developing test plans that are then scripted for execution as part of the Continuous Integration process. The test scripts can be developed once and reused for subsequent CI activities, or they can be developed every time to serve specific testing needs based on the product feature being tested.
Consideration: Consider both Functional and Non-Functional testing using test scripts. Non-Functional aspects can be tested with a more generic test scripting language. Functional testing is very dependent on the complexity and variability of the code.
Tool Usage for Automation
DevOps is characterized by the usage of third-party tools for CI and CD, server configuration, infrastructure provisioning, deployment management, etc. These tools are configured for repeated use without requiring the knowledge of their inner workings. This is not made part of the DevOps engineering stream since the decision criteria will be organization wide.
Consider: Which tools will be used for automation and who will own the decision.
Roles in the DevOps process
There are three high-level processing communities, namely: Production Management; Development and; Automation Engineer. Each of these communities function independently.
A number of articles refer to Global Software Development as a (rapid) production process. Software Development should be viewed as an extension of manufacturing.
The key in production management is recognizing that agile manufacturing is a necessary condition for competitiveness. The agile manufacturing, as discussed in the early 90s, was proposed to solve:
- Products and services with high information and value-adding content
- Mobilization of core competencies
- Responsiveness to social and environmental issues
- Synthesis of diverse technologies
- Response to change and uncertainty
- Intra- and inter-enterprise integration
DevOps adopted the Agile approach, but speed and agility where not the only reasons. Goldman and Nagel argue that; “agile manufacturing assimilates the full range of flexible production technologies, along with the lessons learned from total quality management, & just-in-time production and lean production.” How would we define agility then? Bruce, Daly et al states: “A manufacturing system with extraordinary capabilities (Internal capabilities: hard and soft technologies, human resources, educated management, information) to meet the rapidly changing needs of the marketplace (speed, flexibility, customers, competitors, suppliers, infrastructure, responsiveness). A system that shifts quickly (speed, and responsiveness) among product models or between product lines (flexibility), ideally in real-time response to customer demand (customer needs and wants)”.
From these statements, we can deduce that the Agile approach brings about higher quality products, a reduced delivery time, increased communication between teams, and better morale. These statements align with the agile manifesto principles:
- the software product is the essential focus of the development process
- human behavior and the quality of interactions are essential enablers and success factors
- incremental and spiral approaches based on frequent releases and strict collaboration with the customer, are the quintessential traits of modern software development initiatives
- in general, it is vital to promptly and quickly react to requirement changes at any stage of the development process.
Thus, the launch pad for DevOps adoption is implementing the Agile principles in the software development production process, but the success of an organization goes beyond Agile implementation in ‘IT’ projects. Fitzgerald and Stol states, “Enterprise Agile concept has emerged as a recognition that the benefits of agile software development will be suboptimal if not complemented by an agile approach in related organizational function such as finance and HR.”
The process above describes continuity between Production Management, Development and Automation Engineers. The process clearly shows continuous integration as a practice that emerges to eliminate discontinuities between development and deployment. In the development, community development is represented by the Software Engineering role and deployment is represented by the DevOps Engineering role. The glue that brings together development and deployment and renders it continuous, is the QA Engineering role. In figure 1, a combination of Agile Manufacturing and Continuous Integration results in Continuous Delivery, as depicted by the process. The emphasis is on Continuous Delivery. “We believe that rather than focusing on agile methods per se, a useful concept for assessing continuous concepts in software development comes for the lean approach, namely that of ‘flow’”—Fitzgerald and Stol. Therefore, Agile approach starts the journey of DevOps, but is itself not DevOps. “Rather than a sequence of discrete activities, performed by clearly distinct teams or departments, the argument for continuous software engineering is to establish a continuous movement, which we argue closely resembles the concept of flow found in lean manufacturing and product development, a school of thought that is called ‘Lean Thinking’”. A fundamental focus in lean thinking is that of ‘value,’ and ‘reducing waste’. Any product feature or development step that does not add value is considered to be waste (deploying infrastructure by hand is waste, unwanted, such as unused product features is waste)” – Fitzgerald and Stol.
(Note: From a process architecture perspective, we can identify waste in existing development processes by conducting a value stream mapping exercise, whereby the current processes are identified and visualized to be able to gauge where improvements can be made.)
In this process, the Production Management ‘flows’ into Software engineering. Unlike ‘batch-and-queue’ thinking where actions are done on batches of products, after which they are queued for the next processing step, ‘flow’ refers to a connected set of value-creating actions that occur once a product feature is identified. The Backlog is immediately designed, implemented, integrated, tested, and deployed. This is the essence of the DevOps vehicle. This principle of continuous flow not only refers to a software development function in isolation, but should be leveraged as an end-to-end concept that considers other functions within an organization, such as planning, deployment, maintenance and operation.
(Note: Implementing a purely Agile approach could disguise the software development function as flowing to some degree, but the planning and deployment of features is still done in batches, and not in a continuous flowing movement.)
The proposed ‘flow’ is a continuous process of improvement. The development community will rely on improvements in Application Performance Management, Software Development Processes and Tools for improved implementation of the continuous process and measure the success of the improvements (control) using key process indicators (Metric).
Traditionally, the QA team’s responsibilities consisted largely of mimicking customers’ usage patterns and minimizing the discovery of unpleasant surprises. In the Waterfall model, a QA team’s badge of honour was its ability to anticipate, find, and deliver reproducible scenarios that could result in problems for users. When dealing primarily with client-side code, the variability of client permutations and a lack of accurate behaviour measures make QA input far less objective and verifiable. Today, the more we understand about the code and platform for an application, the greater the chance for building and fixing that application timeously. Therefore, QA Engineering focuses not only on Software testing, but rather on the combination of ‘Operational Acceptance Testing’, ‘Functional Testing’ and ‘Software Testing’.
QA Engineering focuses on software projects that can be labelled as complex adaptive systems (CAS). CAS’s are composed of agents. CAS agents can be molecules, neurons, web servers, fish, starlings, and people – always forming new emergent structures with new emergent behaviors. CAS’s are able to adapt to their environments: an infant learning to walk; a strain of bacteria becoming resistant to an antibiotic; car drivers evading a traffic jam; an ant colony learning about the location of spilled honey; a software team adapting to what their customer really wants. When applying complex systems theory to software development and management, we are treating the organization as a system. From a QA perspective, Quality Assurance Engineering has to focus on both outcome-oriented and process-oriented views to effectively evaluate performance from a user/service satisfaction perspective.
(Note: QA Engineering will rely on new models, e.g. EFQM Excellence Model)
DevOps Engineering is largely based on release engineering, but extends the task by adding automation using a high degree of manage scripting. Thus, DevOps can be defined as, “a set of practices intended to reduce the time between committing a change to a system and the change being placed into normal production, while ensuring high quality.”
Note: The System Testing should not be confused with the QA Engineer tasks. This testing is merely ‘verification for readiness to proceed based on the QA Engineering documentation’.
Automation Engineering is placed in its own community. Automation Engineering is an enterprise function— “Automation engineers design, program, simulate and test automated machinery and processes in order to complete exact tasks”. This enterprise function focuses on the tools that are needed by the Development community, as well as the Test Plan and test artifacts (scripts, etc.). This community’s role should not be confused with the tasks of a DevOps engineer. The automated machinery in this discussion is the end-to-end DevOps pipeline. The Automation Engineering group is responsible for putting in place the Enterprise DevOps pipeline. This pipeline is based on organization and industry standards.