What is Quality Assurance

“the managing of every stage of a production process to make certain that the goods being produced are of the intended standard” – definition of Quality Assurance, Cambridge Dictionary

Any business or organization which produces services or products wants to ensure that they maintain the expected quality for their customers. Products and services which are technology-based rely on the architecture to provide the foundation or framework for design. Therefore, it is important to ensure that the architecture meets the expected quality since the following designs and implementation will be based on the architecture.

Quality assurance (or QA) is the process by which we can ensure that an architecture meets the expected quality standards. This process should check both the process by which the architecture is created as well as testing the resulting architecture.

Quality Assurance is commonly practiced through two methods that attempt to ensure that the architecture meets the quality expectations, validation and verification. 

Why do we need Quality Assurance

Perhaps the most important reason for quality assurance is that the customer receives a product or service at the expected quality. Without quality assurance it is difficult to assess the level of quality before the product or service reaches the customer. Low-quality deliveries to the customer eat at trust, and damage business reputation. This has a significantly negative effect on business. Since most technology products and services are built upon an architecture it is of critical importance that the architecture maintains a good quality.

Performing quality assurance throughout the architecture and development process helps to find design defects and errors early in the development cycle. Defects which are found early in the cycle are less expensive to fix and require less effort.

Quality assurance supports the development of architecture to the right quality, where quality is often a trade-off with time and cost. When developing an architecture it is important to ensure the right balance of quality, and ensure that the architecture is fit for its intended purpose. Performing good quality assurance practices supports getting the balance right.

Quality assurance applies not only to testing the quality of the end-product but also to the process by which the product is developed. This also applies to the process by which we ensure quality in the architecture. This is particularly important in industries that have to comply with strong regulatory requirements and governance. Quality assurance supports governance ensuring that standards and regulations are followed.

Quality Assurance Approach

Quality Assurance Throughout the Whole Process

Quality assurance is something that is performed throughout the whole process of developing an architecture. Don’t leave it to the end of product development. Starting quality assurance early and continually checking the architecture will lead to detecting ineffective design, errors and problems early in the process, making it less expensive to fix.

Architects Outside the Team

Involve architects, who are outside the team, in the quality assurance process. Architects who work with an architecture day-in-day-out can become blind to design defects. Involving architects from outside the team can often result in detecting defects much faster and even the discovery of new ways of improving quality.

Cost and Quality Assurance

Quality Assurance costs effort and time to operate, and can also cost products in terms of time-to-market. It is important to get the right balance of quality assurance against cost, just enough to ensure the expected quality. Some organizations may feel that an easy way to cut costs is to reduce expensive QA activities, however this can have significantly negative effects on business.

“If you think that Quality Assurance is expensive, wait until you see the cost of poor quality.”

Wasting Time

The effect of poor quality is the expenditure of effort on non-value activities, for example, correcting mistakes, provision of extra support, appeasing unsatisfied customers, or performing ineffective workarounds.

“Time waste differs from material waste in that there can be no salvage. The easiest of all wastes, and the hardest to correct, is the waste of time, because wasted time does not litter the floor like wasted material.”

Henry Ford, Ford Motors

As Henry Ford points out, waste of time cannot be regained, it is lost without gaining any value. Good quality assurance practices help to avoid, correct and facilitate effective use of time.

Verifying the Architecture

Verification of the architecture asks the question: “Are we building the architecture right?”. Verification of the architecture comes before validation. Verification attempts to ensure the quality of the architecture before the construction of the product or service based on the architecture.

Architecture Way of Working 

One way to ensure quality in the architecture is to have well-defined processes and activities for developing the architecture, and check that these processes are followed. This provides clarity regarding the way of working and ensures that the architects have a common practice. Weaknesses in quality may appear if practices are unclear or missing. 

It is also important that the way of working defines roles and responsibilities, this clarifies responsibility and accountability for the various activities and deliverables.

Architecture Deliverables

Defining expected architecture deliverables provides an effective method of detailing expectations from the architecture work, and provides underlying support for reviews and inspections. Checklists provide a simple method for ensuring that all required deliverables are present when the architecture is delivered.

The definition of a deliverable shall describe the expected contents of the deliverable, for example, expected topics and views of the architecture.

Reviews and Walkthroughs

Reviews and walkthroughs are a great way to ensure quality in the architecture before it is published for use. These allow the architect to explore designs, gain feedback and improve the quality of the architecture. There are many different types of reviews, and the reviews an organization puts in place will depend on the way the organization works and the business context. Reviews generally involve several stakeholders, architects, product owners, lead developers, etc…. The following sections describe some different types of reviews.

Content review

A content review focuses on the content of the architecture. These types of reviews often focus on a specific design aspect of the architecture or are used to evaluate selected design alternatives. These reviews focus on the quality of the architecture, to ensure that it is “fit for purpose”.

These types of reviews can be used to evaluate different aspects of the architecture, for example,  significant functional design, quality attributes, information models, solution context or explore a concept.

Content reviews can be conducted formally with a protocol based on the expected content of a given deliverable, or they may be informal as a way of verifying ideas or concepts during the architecture development. It is recommended that content reviews be performed regularly throughout the development of the architecture.

Way of working review

Way of working reviews focus on the process and activities by which the architecture is developed. This review aims to continually improve architecture practices to ensure that they are effective and promote quality. The aim of such reviews is to assess if the current way of working is resulting in good quality architectures, and to explore what can be done to continually improve quality.

Note that there doesn’t have to be a specific quality problem to hold a way of working review, this should be a continual effort to improve quality.

Delivery review

A delivery review is often a formal review performed before publishing an architecture, or a significant change to an architecture. This type of review is often used to check that all required architecture deliverables are complete and that all required content is present before publication to stakeholders. This type of review is often performed to a protocol or checklist, and can even consider aspects of the deliverables such as language, document formats, coherence, correctness and readability.  

Peer-reviews

An architect can continuously review an architecture against standards and industry methods, however, it is often more effective to review architecture decisions and concepts with other architects. Reviewing an architecture with a collective group of architects will broaden both the architecture competence, provide different perspectives and gain a broader agreement for concepts. Performing peer-reviews regularly leads to stronger architecture solutions.

External-reviews

Using external parties to review an architecture provides an independent assessment. This can be a useful technique to counter any architecture bias within an organization, and raise questions that the internal organization simply has not considered. An external review, if performed by persons external to the organization, can also approach politically sensitive or cultural issues which are perhaps difficult to address by the internal organization. External reviews are often not as frequent as peer reviews, they are used at the beginning of a large assignment to validate the direction of an architecture, or close to architecture delivery to ensure quality.

Modelling Standards

The provision of common models and standards within architecture supports the reuse of tried and tested design methods. Using common models provides a common language for describing the architecture, this makes the architecture understandable to other architects and stakeholders. UML and Archimate are examples of such modelling languages. Providing a standard set of rules for describing an architecture makes the architecture easier to review and improves quality.  

Automated Architecture Analysis

Some modelling tools provide analysis and simulation methods to analyze architectures. This is a useful way of testing architecture designs before implementation. This method may be used to analyze and simulate processes that carry significant risks if they fail, or processes that are expensive to implement and change. The use of analysis and simulation means that errors and defects can be identified at an early stage.

Validating the Architecture

Validating the architecture asks the question: “Are we building the right architecture?”. Validation of the architecture comes after verification. Validation tests the effect of the architecture after construction of the feature, product or service. Since architecture is mainly a planning activity for implementation, the emphasis is perhaps more on verification, however even after implementation it is important to test that the architecture meets the requirements.

Proof of Concepts (POC)

One way to test an architectural concept is to make a prototype implementation and see if it works. This is generally referred to as a Proof of Concept, or POC. This is a very common way to explore and test architecture designs and measure desired effects. While this is not the same as a production-quality implementation, it does reduce risk. A key principle with POCs is that they are experiments, and should not be used as if they are production-ready code. After validating the POC, the implementation will likely need to be re-written for production quality implementation.

Requirements Validation

Once a product or service is implemented and executable, the requirements on the architecture can be validated. Quality attributes provide a set of non-functional requirements on the architecture, but these can still be tested. For example, performance and robustness as quality attributes are often tested before product release.

Significant functional requirements on the architecture are also validated through functional testing. This may be tested and validated through functional tests on the product or service. For example, a functional test for logging in the product may act as a verification of a logging requirement in the architecture.

The architecture may be required to meet regulatory requirements and it is often very important to validate these requirements after implementation.

Inspections

Inspections are related to governance and may perform detailed checks that procedures and processes were followed during development, as well as checking that a product conforms to a given standard. Inspections are often carried out close to the final release of a product since they are detailed and time-consuming. An inspection may be carried out by an internal part of the organization, but they are often carried out by a regulatory body if a product is under regulation, for example, FDA (Food and Drug Administration, USA), or NRC (Nuclear Regulatory Commission, USA).

Measure Effect and Outcomes

Even after the launch of a product or service, it is important to measure the effect of the architectural change. Expected outcomes are often detailed in objectives, these can be measured to assess if the architecture is performing as expected. Rather than just validating the architecture against requirements, this validates that the architecture is delivering value.  

References and Further Reading

Validation and Verification
https://www.geeksforgeeks.org/software-engineering-verification-and-validation/?ref=gcse  

UML
https://www.uml.org/ 

Archimate
https://pubs.opengroup.org/architecture/archimate3-doc/