API full-scenario zero-code testing robot, Huawei Cloud releases ATGen in CodeArts TestPlan

As we all know, the interaction between software services and components mainly relies on a large number of API interfaces.

Take Huawei Cloud's more than 300 commercial cloud services as an example. Each service contains an average of 500 interfaces, and the total number of interfaces is as high as 100,000. The interface calling context business scenarios are unimaginably complex.

How to ensure that the API test scenario is as complete as possible, without omissions, or causing serious problems on the live network? This is a classic problem that people currently encounter, and the common bottlenecks are mainly as follows:

• The current interface test design relies on personnel experience, and the test data construction method is single, which is difficult to reflect the actual application data distribution scenario of the existing network;
• Faced with thousands of API interfaces, which can constitute tens of thousands or even hundreds of thousands of operation paths, the E2E scenario test and orchestration workload is huge;
• Existing API interface test generation tools basically use Fuzz testing based on SBST guided by code coverage goals, or single-interface playback based on live network data. They are not aware of the interface business context. For multi-interface function interaction scenarios, the generation effectiveness Poor and inefficient.

So, how does Huawei Cloud solve these problems?

▶Huawei Cloud ATGen: Perceive the API interface context, independently generate API full-scenario tests, and transform the human-computer interaction model

ATGen (APITestGenerator) is a context-aware API scenario-level zero-code independent test generation service that realizes fully automatic design, generation, execution and judgment of application API scenario-level tests without writing a single line of code.

The traditional test design and execution interaction model requires manual understanding of design documents, arrangement of test plans, and test automation development. This requires participants to be familiar with product or service business and interface documents, and to be proficient in testing technology, test automation framework, and test automation. Code writing.

Nowadays, with the support of AI algorithms that are independently designed, generated, executed, and judged independently, the human-computer interaction model of API scenario-level testing has undergone tremendous changes.

• ATGen human-computer interaction mode supported by intelligent algorithm

During the testing process, the machine independently parses documents and data and automatically generates an API scenario-level operation dependency ODG (OperationDependencyGraph) graph. The tester only needs to make appropriate revisions, and the machine can autonomously explore and traverse the ODG graph execution and complete the results synchronously. For judgment and report generation, testers only need to confirm the results in batches by category.

▶Huawei Cloud ATGen: Highlighted Technology and Business Value

Based on the forward API interface definition document, Huawei Cloud ATGen automatically mines the API test context operation dependency graph (ODG), explores and traverses the ODG to independently generate API test sequences, constructs and issues API test requests in real time, and determines the API test response results. Dynamically correct the ODG map and optimize the next round of generation.

Six key features of Huawei Cloud ATGen:

• Supports fully automatic intelligent test generation based on the Rest API interface definition of Yaml document zero code;
• Perception of the Rest API interface call context: Parses the interface definition Yaml document, mines the context parameter transfer dependencies between API interfaces, and CRUD addition, deletion, modification and query dependencies, and generates the interface operation sequence dependency graph ODG ;
• Autonomous exploration, traversal and correction of ODG: Explore and traverse ODG to generate test sequences in batches, automatically issue them for execution, and dynamically correct the ODG diagram based on response feedback;
• 13 interface test data generation methods: Support interface context automatic parameter passing, multiplexing dictionary values, enum values, example values, example variation values, boundary values, intermediate values, random values, etc. 13 A test data generation method;
• Automatic mining to generate test decision points: Generate explicit test decision points based on interface definition and status code;
• Test result clustering and batch efficient confirmation: Supports hierarchical clustering of test results according to business return codes, similar subsequences, and parameter generation types, making it easier for testers to confirm failures in batches by category and report them together. Key bill of lading;

*For specific technical details, please refer to the top conference papers published in ICSE and ASE [1, 2].

Huawei Cloud ATGen business value and application prospects:

• Quickly build an interface and functional quality protection network: For newly established product service teams, or full-featured teams without full-time testers, zero-code full automation with little or no participation can be achieved Full scenario-level testing of interfaces to quickly build a basic quality protection network for interface-level and functional scenario-level testing;
• Low cost, high coverage: For a test team with full-time testers, test experts, and high maturity, test design efficiency can be greatly improved and higher test scenario coverage can be achieved at low cost. and defect interception rate;
• Pipeline quality access control reinforcement: Can be integrated with existing pipelines to reinforce existing smoke test access control and further intercept business logic deep defects;
• Uncoded interface-oriented full-scenario and full-attribute testing: Can realize fully automated generation of abnormal scenarios (reliability testing) and concurrent scenarios (performance pressure model) based on normal API scenario testing, completely Replaces the existing interface fuzz test and implements zero-code testing of the interface in all scenarios.

Huawei Cloud ATGen has deployed 30 daily applications of products and services within Huawei, and measures business application effects in three dimensions:

• Generation effectiveness: Including the successful pass rate of interface requests, the longest sequence length and length distribution of successful requests, the number of use cases adopted and baselined by the business, etc.;
• Number of defect detections: Defect interception capability, that is, the number of problems and false alarm rate of product and service defects found;
• Coverage: Includes white-grey-black box coverage, that is, code coverage, interface and parameter combination coverage, business scenario coverage, etc.

Three typical business application scenario practices within Huawei

• Business application scenario 1

Product form: storage management & control service cloud products, northbound service-based, southbound control of embedded storage devices

Test team size: 50 people

Total number of northbound Restful interfaces: 2300

Current situation before application: The team is newly established and lacks interface and functional test automation protection network

Application mode: quickly build interface test and functional test protection network test cases from scratch

Application effect: A total of 350 interface defects were found, covering 3 major categories of defects. The effectiveness of API interface level requests can reach 80%, and the false positive rate is 10%.

• Business application scenario 2

Product form: operator & enterprise network assurance & intelligent operation and maintenance cloud services, public cloud/private cloud deployment

Test team size: 50 people

Total number of northbound Restful interfaces: 10,000

Current status before application: 8000 interface & functional test cases

Application mode: coverage enhancement and supplement to existing test cases

Application effect: Scenario test coverage increased by 30%, and 30 new deep business logic problems were discovered. Examples are as follows:

• Business application scenario 3

Product form: XX tool cloud native service

Test team size: full-featured team, no full-time testers

Total number of Restful interfaces: 1300

Current status before application: 10,000 interface test cases

Application mode: Integrate ATGen into the tool service alpha and beta environment pipeline. After the build is completed and the regression test task is completed, the generation task is automatically triggered to detect whether there are failed interfaces as alpha and beta access control reinforcement.

Application effect: 60 interface defects were additionally discovered, and the effectiveness of API interface test generation can reach 82%.

Faced with the rapid iteration of cloud product versions, application scenarios are becoming increasingly rich, and product functions are becoming increasingly complex. It is becoming increasingly difficult to rely solely on human testing experience to ensure quality. The independently generated full-scenario functional test robot ATGen emerged as the times require. It can achieve low-cost, zero-code, high-coverage, and low-false-positive API full-scenario testing, and continue to provide developers with high-quality services. 【1】"MOREST: Model-based RESTful API Testing with Execution Feedback", ICSE ’22,

【2】“Industry practice of automatic RESTful API testing”, ASE’22

Follow @HUAWEICloud for more information

The above is the detailed content of API full-scenario zero-code testing robot, Huawei Cloud releases ATGen in CodeArts TestPlan. For more information, please follow other related articles on the PHP Chinese website!