Summary : This article explores the underlying logic of software testing, defining it as the process of obtaining comprehensive quality information based on real user requirements. It addresses the "Why, What, and How" of testing and emphasizes that testing is an endless, probabilistic sample experiment aimed at mitigating quality risks.

What is the Underlying Logic of Software Testing? (Part 1)

What is the underlying logic?

According to teacher Liu Run's explanation: “The common ground between things is the underlying logic. The underlying logic is something that represents only the similarities among differences and remains unchanged behind the changes. ...... Underlying Logic + Environment Variable = Methodology”

He also said: "Only the underlying logic is alive." So, let's explore: what is the underlying logic of software testing?

1. Basic Understanding of Software Testing

A basic understanding of software testing helps us reach a consensus. Based on this consensus, it is easier to discuss the underlying logic of software testing. The basic understanding of software testing can be described in one sentence, which grasps the essence of software testing and succinctly outlines correct software testing values. Rather than being an individual’s personal values, these are values accepted by most people in the field.

In Chapter 1 of Full Software Testing (Version 3), we discussed the cognition of software testing in depth:

Software testing verifies whether the functional characteristics of software meet the requirements.
Software testing finds defects in the software.
Software testing includes static testing—requirement, design, and code review.
Software testing systematically and completely evaluates the quality of software products and provides quality information.
Software testing exposes and reveals product quality risks.
Software testing is not only a technical activity but also a comprehensive social and psychological activity.
Software testing greatly reduces the cost of poor quality by investing in quality assurance.

According to the list above, the basic cognition of software testing can be explained in one sentence: Based on the understanding of the real needs of users, obtain true and comprehensive quality information about software products through various means. Whether verifying if software features meet requirements, evaluating product quality, or revealing quality risks, judgments are made based on the real quality information obtained. Defects can be regarded as a by-product of this activity.

This emphasizes the understanding of real user requirements. On one hand, it reflects that "there is no quality without users, and quality exists relative to users." We must conduct tests from the perspective of users. On the other hand, it refers to real user requirements, not false or incorrect ones. Business requirements should eventually be decomposed into user role requirements, and the functional/non-functional requirements of the system are also designed to meet user needs.

The "software product" mentioned here is not limited to executable programs but also includes data, requirement documents, design documents, source code, user manuals, technical manuals, etc.

After understanding "what software testing is," we can discuss its underlying logic.

2. The Underlying Logic of Software Testing

The underlying logic of software testing can be summarized by answering three questions:

Why?
What?
How?

Moreover, in the process of answering these three questions, our answers should adapt to different test objects (such as Windows/macOS native applications, web software, mobile apps, and embedded software), different test types (such as functional, performance, security, and compatibility testing), different test levels (such as unit, integration, and system testing), and different teams and products. Therefore, the answers become universal. Although different test methods, technologies, and practices exist in different contexts, we can abstract their common ground.

Based on this consideration, let's answer these three basic questions:

Why? As long as the work is done by humans, there is no guarantee of infallibility, and problems will occur. If software is released with problems, it is likely to cause losses to customers or lead to dissatisfaction, eventually damaging the enterprise's interests. Numerous past quality accidents have proven this. Before delivery to customers, software needs to be fully tested; otherwise, there are serious consequences.
What? This depends on the delivery quality objectives. Starting from these objectives, they are decomposed, and then the quality data to be obtained about the test object is determined for each specific sub-objective, thereby establishing the scope of the test or the test items. Going further, we determine the priority of test items according to users' varying sensitivities to quality characteristics and functional features. This section belongs to test analysis and involves testing risks and strategies.
How? We find ways, methods, or means to obtain the quality data of the tested object, including test scheme design, scenario design, test cases, or test data design.

That is: For Quality, from Quality objectives, and by getting Quality data.

I previously wrote an article titled Three Questions About the Soul of Software Testing and How to Answer Them. Do the answers to these "Three Soul Questions" also reflect the underlying logic of software testing?

Q1: Why can't all bugs be detected?
Q2: How do we test? Are the testers professional enough?
Q3: Hurry up! Why do the testers always hinder the whole team and report bugs at the very end?

In fact, they reflect another layer of the logic of software testing:

The answer to Q1 presents this underlying logic: Testing is an endless process, testing is always full of risks, and the more bugs a developer writes, the more bugs the test misses. We can only prove that the defects we found exist; we cannot prove that there are no defects in the software. This is because testing is a sampling experiment.
The answer to Q2 presents this underlying logic: The testing work (including the formulation of test objectives, the process of test analysis, and corresponding test design methods) can be clearly explained, and testing is not an isolated job. Due to the influence of requirements (e.g., fuzzy requirements), system design (e.g., coupling and complexity), and programming (e.g., secretly modifying code), testing must cooperate closely with product management and development.
The answer to Q3 presents this underlying logic: We can complete test analysis, test planning, and test design before developers finish writing the code. However, to actually run system tests, we must wait until developers finish the builds. Test execution is a later-stage task, and testing time is easily squeezed by earlier development delays. Therefore, project delays easily create the illusion that testing is lagging behind.

The Underlying Logic of Testing (Probabilistic Thinking): A test is a sample experiment that requires careful analysis and design to reveal quality risks at the earliest possible opportunity with minimal effort. Since it is a sample experiment, defect distribution follows a normal distribution, and quality can be improved from 3-sigma to 6-sigma, but it will never reach 100%.