Testing Your Tests? Who Watches the Watchmen?

Whether you are working for a large enterprise, a startup or yourself, unit testing is not only useful, but is often indispensable. We use unit tests to test the code, but what if our tests are wrong or incomplete? What can we use to test our tests? Who will supervise the inspector?

Key Points

• Variation testing is a technique that evaluates its quality with a small number of modified tests and can be used to test the test itself. It involves creating a "variant" or variant of the original test and checking if these changes are detected by the test.
• Humbug is a variant testing framework for PHP that can be used to generate code coverage. However, it is PHPUnit-specific and can be problematic for users using different testing frameworks.
• Although variant testing is valuable, it also has its disadvantages, mainly reflected in performance. This is a slow process because it relies on many factors such as the interaction between lines of code, the number of tests, the level of code coverage, and the performance of code and tests.
• As the complexity of the application increases, the importance of maintaining 100% code coverage increases. Tools like Humbug are critical to maintaining such coverage, especially in the enterprise ecosystem.

Mutation test

No, it's not that kind of mutation. Variation testing (or Variation analysis) is a technique used to create and evaluate the quality of software tests. It involves modifying the test in a very small way. Each modified version is called a variant, and the test detects and rejects variants by causing the original version to behave differently from the variant. Mutations are errors in the original code, and analysis checks whether our tests detect these errors. In short, if the test is still valid after the mutation, it is not a good test.

Mutation test using Humbug

Humbug is a variant testing framework for PHP. In order for Humbug to generate code coverage, we must install and enable XDebug on our machine. We can then install it as a global tool.

composer global require 'humbug/humbug'

After

, if we run

humbug

The

command should be able to see some Humbug installation information and an error indicating that we do not have a humbug.json file.

Boot program

Before we configure and use Humbug, we need a project that can be tested. We will create a small PHP calculator package where we run our unit tests and mutation tests. Let's create a /Calculator folder. In it, let's create our /src and /tests folders. In our /src folder, we will have our application code; the /tests folder will contain our unit tests. We also need to use PHPUnit in our package. The best way is to use Composer. Let's install PHPUnit using the following command:

composer global require 'humbug/humbug'

Let's create our calculator. In the /src folder, create a Calculator.php file and add the following:

humbug

This is a fairly simple program. A simple calculator with basic arithmetic, percentage and logarithmic operations and functions that return π values. Next, in our /tests folder, let's create unit tests for our calculator. If you need help with unit testing in PHP, check out this tutorial. Create a CalculatorTest.php file and add the following:

composer global require phpunit/phpunit

This will be our initial test stack. If we run the phpunit command, we will see it execute successfully and our 4 tests and 4 assertions will pass. It is important that all tests must be passed, otherwise Humbug will fail.

Configuration Humbug

Humbug can be configured manually by creating a humbug.json.dist file, or automatically by running the following command:

<?php
namespace package\Calculator;

class Calculator {

    /**
     * 基本运算
     */
    public function add($a1, $a2) {
        return $a1 + $a2;
    }

    public function subtract($a1, $a2) {
        return $a1 - $a2;
    }

    public function multiply($a1, $a2) {
        return $a1 * $a2;
    }

    public function divide($a1, $a2) {

        if ($a2 === 0) {
            return false;
        }

        return $a1 / $a2;
    }

    /*
     * 百分比
     */

    // 这将返回 a1 的 a2 百分比
    public function percentage($a1, $a2) {
        return ( $a1 / $a2 ) * 100;
    }

    /*
     * π
     */

    // 返回 π 的值
    public function pi() {
        return pi();
    }

    /*
     * 对数
     */

    // 返回以 10 为底的基本对数
    public function log($a) {
        return log10($a);
    }

}

Running this command will ask us to answer some questions:

• What source directories do you want to include? Here we will use src/, which is our source code directory.
• What directories do you want to exclude from the source directory? can be useful in some cases, such as the external vendor directory we do not want to test. It does not apply to our current situation.
• Single test suite timeout (seconds). Let's use this for 30 seconds. This may be too much, but we want to make sure everything has enough time to run.
• Where do you want to store the text log? humblog.txt as the default value, we will keep it.
• Where do you want to store the json logs (if needed)? The default value is empty, but we will store it in humblogjson.json.
• Generate "humblog.json.dist"? After this file is generated, it will contain all the configuration values ​​we just provided. If we want to change something, we can edit it manually.

Use Humbug

Now that we have run the application and tested it, and we have Humbug installed, let's run Humbug and check the results.

<?php
use package\Calculator\Calculator;

class CalculatorTest extends PHPUnit_Framework_TestCase {

    public function testAdd() {
        $calculator = new Calculator();
        $result = $calculator->add(2, 3);
        $this->assertEquals($result, 5);
    }

    public function testSubtract() {
        $calculator = new Calculator();
        $result = $calculator->subtract(6, 3);
        $this->assertEquals($result, 3);
    }

    public function testMultiply() {
        $calculator = new Calculator();
        $result = $calculator->multiply(6, 3);
        $this->assertEquals($result, 18);
    }

    public function testDivide() {
        $calculator = new Calculator();
        $result = $calculator->divide(6, 3);
        $this->assertEquals($result, 2);
    }

}

The result should be close to this:

Interpretation of Humbug Results

The number of mutations created is just the number of small changes introduced by Humbug to test our tests. The killed mutation (.) is the mutation that causes the test to fail. Don't be confused, this is a positive result! Escape mutation (M) is the mutation that still passes the test. This is not a positive result, we should go back to our tests and check what is missing. Uncovered mutation (S) is a mutation that occurs in rows that are not covered by unit tests. Fatal error (E) and timeout (T) are the mutations that create fatal error and create infinite loops, respectively.

What about the indicators?

Variation score metrics represent the percentage of mutations detected. Our goal is 100%. Variation code coverage represents the percentage of tests covered by the mutation. Variation score metrics can give you an idea of ​​the effectiveness of existing tests. Analyzing our humbug logs, we can see that we have 9 uncovered mutations, as well as some very bad metrics. Check out the humblogjson.json file. This file is automatically generated like the humblog.txt file and contains more detailed information about the causes, location, and reasons for the failure. We did not test our percentage, π, and logarithmic functions. Additionally, we need to cover the case where the number is divided by 0. Let's add more tests to cover the missing situation:

composer global require 'humbug/humbug'

This time, 100% means that all mutations are killed and we have full code coverage.

Disadvantages

The biggest disadvantage of mutation testing and Humbug is performance. Variation testing is a slow process because it relies on many factors such as interactions between lines of code, number of tests, code coverage level, and the performance of code and tests. Humbug also performs initial test runs, logging, and code coverage, which increases the total duration. Additionally, Humbug is PHPUnit-specific and can be a problem for users using other test frameworks. That is, Humbug is under active development and will continue to improve.

Conclusion

Humbug can be an important tool for maintaining the life of your application. As application complexity increases, so does testing complexity—and it becomes very important to always keep 100% testing, especially when dealing with the enterprise ecosystem. The code used in this tutorial can be cloned here. Have you used Humbug? Are you doing mutation testing in other ways? Please tell us all your thoughts on this!

Frequently Asked Questions about "Who will supervise the inspector?" (FAQ)

What is the origin of this sentence "Who will supervise the inspector?"

"Who will supervise the inspector?" This sentence originated from the Latin phrase "Quis custodiet ipsos custodes?", created by the Roman poet Juvenal. This sentence is often used in discussions that question the integrity and accountability of those in power. It's basically asking, "Who will guard the guard?" or "Who will monitor the people who monitor us?" This sentence has appeared in various forms of media, including Alan Moore, Dave Gibbons and John ·Higgins' popular graphic novel series "Watchman".

In the context of software testing, how is the sentence "Who will supervise the inspector?" used?

In the context of software testing, "Who will supervise the inspector?" is a metaphorical question that solves the reliability and accuracy of the test. It questions who or what is monitoring the tests to ensure they can function properly and produce accurate results. This is a key aspect of software development because it ensures the quality and reliability of the software being developed.

What is the importance of testing and testing in software development?

Test testing, also known as test verification, is a key part of software development. It ensures that testing accurately measures the functionality and performance of the software. If there is no test verification, there is a risk that the test may produce false positives or missed reports, resulting in an inaccurate assessment of software quality and reliability.

How do I make sure my tests are reliable and accurate?

Ensure the reliability and accuracy of the test involves several steps. First, you should thoroughly check your tests to make sure they are designed and implemented correctly. Second, you should verify your tests regularly by comparing the test results with known results. Finally, you should continuously monitor and update your tests to ensure they remain accurate as the software evolves.

What are the common pitfalls in software testing?

Some common pitfalls in software testing include insufficient testing, wrong testing, and not understanding the purpose of testing. Other pitfalls include over-reliance on automated tests without understanding their limitations, and periodic review and update of tests.

What is the relationship between the graphic novel "Watchman" and software testing?

The graphic novel "Watchman" uses the sentence "Who supervises the inspector?" to question the responsibility and integrity of those in power. In the context of software testing, this sentence can be used to question the reliability and accuracy of the test itself. Just as the Watcher should protect society, testing should protect the quality and reliability of software. But just as the Watcher needs to be monitored, the test needs to be monitored.

What is the role of a software tester?

The role of software testers is to ensure the quality and reliability of the software by designing and implementing testing. These tests are used to identify and fix errors, verify functionality, and evaluate performance. Software testers must also monitor and update these tests to ensure they remain accurate during the software development.

How do I improve my software testing skills?

Improving your software testing skills requires continuous learning and practice. You should be aware of the latest testing methods and tools and practice designing and implementing tests regularly. You should also seek feedback on your tests and be happy to learn from your mistakes.

What are some good resources to learn more about software testing?

There are many resources available to learn more about software testing. These include online courses, books, blogs and forums. Some recommended books include "The Art of Software Testing" by Glenford J. Myers, "Software Testing: The Craftsman Method" by Paul C. Jorgensen, and Seim Karnar and Jack Falke "Testing Computer Software" by Huang Q. Ruan.

What is the future of software testing?

The future of software testing may be strongly affected by technological advances. This includes the increasing use of automation and artificial intelligence in testing, as well as the development of new testing methods to adapt to emerging technologies such as virtual reality and blockchain. However, the basic principles of software testing—to ensure the quality and reliability of the software—will remain the same.

The above is the detailed content of Testing Your Tests? Who Watches the Watchmen?. For more information, please follow other related articles on the PHP Chinese website!