Quality model and practice of Meituan’s comprehensive business recommendation system

Author: Yong Haogengen, Wang Xin, etc.

1 Foreword

Meituan’s in-store comprehensive business (hereinafter referred to as the comprehensive business) is Meituan’s in-store comprehensive business One of the important sectors of the in-store business, covering dozens of key sub-industries such as bathing, KTV, beauty industry, medical beauty, parent-child, marriage, sports and fitness, entertainment, education and training, home furnishing, pets, bars, life services, etc., to meet the needs of The diverse local life needs of hundreds of millions of users.

The recommendation system is an important link in achieving efficient matching of supply and demand, and is an outlet for transmitting data value. The quality of the recommendation system determines the loss of the matching effect. As shown in Figure 1 below, the data is processed by the data warehouse and algorithm, and then sent to various business systems through data services. Finally, it is transferred back to the data warehouse through the client, forming a "flywheel effect" of data, and quality is exactly this. The key point of gear meshing in a link is an important prerequisite for improving efficiency and ensuring effectiveness.

Quality assurance must be carried out around measurement so that it can be "visible", "clarified" and "corrected accurately". However, traditional backend service quality indicators cannot well describe the quality of the current "data flywheel". We hope that through the construction of a quality model for a comprehensive business recommendation system, we can provide a new thinking perspective and practical reference for similar multi-business line, effect-oriented system quality measurement.

Figure 1 "Data Flywheel" of the recommendation system

2 Current Situation Analysis

The recommendation system is an effect-based system, and its quality characteristics are different from those of functional systems. Generally, functional systems will significantly affect user experience after being downgraded. However, if the recommendation result returns A or A', it is difficult for users to notice it clearly. But in fact, if the matching effect becomes worse, it will directly affect the user's implicit experience and needs to be identified. Functional systems generally build quality index systems with usability as the core. In the business practice of comprehensive business recommendation systems, we found that usability and other indicators have the following limitations:

• Availability is not sensitive to some defects: Availability is a function of interruption frequency and duration, which reflects the system's ability to continue to provide services. As long as system defects do not affect external services, they do not affect usability, but some actually affect user experience. The defects here may be expected (such as active downgrade) or unexpected (model update delay), and should be included in the measurement of quality.
• Availability is difficult to cover the entire link of data: The link of the recommendation system covers data production, processing, application, analysis and other links. One is that usability does not involve the quality of the data table, and the other is that where performance measurements are available, they cannot reflect the full picture of data quality. Data quality needs to consider characteristics such as completeness, accuracy, timeliness, and security, which goes beyond the scope of availability. Internationally renowned scholar Andrew Ng once said that 80% of the value of artificial intelligence depends on data, and the quality of the recommendation system’s delivery of recommendation effects (click conversion rate, transaction conversion rate, user stay time, etc.) also mainly depends on Data quality.
• Usability is difficult to reflect business differences: Meituan Zhizong covers hundreds of industries and dozens of channel pages. The recommendation system is based on efficiency and cost considerations. Businesses cannot be completely isolated, and the series-parallel calculation method of availability makes it difficult to differentiate between businesses for separate evaluation. Different services on the network vary greatly, with different access frequencies, peak traffic periods, and business strategies. As a result, quality characteristics and problem distribution are also different. Current availability indicators lack business dimension information, which is not conducive to guiding refined quality operations.

In quality construction, the fault level was used as the target in the past. The verification cycle was long and contingency, and the relationship between the target and the logical derivation of actions was not strong. In addition, the fault itself is an afterthought, and this problem-driven approach is not conducive to continued operations. In general, with usability as the goal, there are various problems in actual implementation of calculations, so we consider building a quality model for the recommendation system, based on usability, and then adjusting the calculation method to guide refined quality operations.

3 Construction Ideas

3.1 Quality in the Business Context

To build a quality model, first return to the understanding of the essence of quality. According to the definition of the International Organization for Standardization (ISO), quality is the sum of characteristics that reflects an entity's ability to meet explicit or implicit "needs." Another commonly used quality concept is stability. The core of stability is to allow the system to run in the "expected" state for a long time. Whether it’s quality or stability, it’s important to understand whose needs and expectations the system needs to meet. In the recommendation scenario, this object is the product and algorithm. Business products understand user scenarios, abstract user needs, and propose product requirements to the recommendation team, which is reflected in external product iterations. At the same time, the recommendation system team collaborates with each other internally to learn the best optimization model strategy, which is reflected in algorithm iterations within the data team.

As shown in Figure 2 below, in the calculation formula of availability, the long time is emphasized, while "need" and "expectation" are only reflected in the provision of external services. There is a certain rationality here. First, availability is a common indicator in the industry, and the definition must be generalized. Then the commonality and bottom line of quality is the provision of external services; second, most back-end systems deliver functions, and most external services are provided in the "have" "Between" and "none", there is also some room for service degradation. However, for recommendation systems with effect as the core goal, there is a long spectrum of "good" and "bad" effects between "with" and "without" functions. In our iteration of thinking about the quality of the recommendation system, the core change is from the "yes" and "no" of external services to the "good" and "bad" of external services. This is also the starting point for transforming the usability calculation method.

Figure 2 The cognitive impact of defects on quality metrics

3.2 Defect consideration and selection

Failure to meet "needs" or "expectations" will cause defects, and defects are the cause of quality loss. ISO/IEC 25010 Software Quality Model (2011) The software quality model defines software defects and can be regarded as the complete set of defects. It includes functional suitability, performance efficiency, compatibility, availability, reliability, security, and maintainability. , 8 features and 31 sub-features of portability. There are some quality features that are not covered by backend services (User interface aesthetics, ease of learning, etc.), and there are some outstanding elements that do not constitute C-end quality in current cognition (Modularity, coexistence , non-repudiation, reusability, etc.). Combining the business characteristics and high-frequency quality issues of the recommendation system, at this stage we focus on considering the quality characteristics as shown in Figure 3 below as the source of defects.

Figure 3 Quality characteristics of the recommendation system

We found that traditional usability measures are mostly concentrated in reliability, functional completeness, and correctness, but there is a lack of measurement for most functional accuracy, adequacy, and safety, which are closely related to the quality and effectiveness of recommendations. The impact of accuracy and appropriateness on the effect is relatively intuitive, while others are more indirect. For example, in security, take crawler access in security as an example. Because the access behavior of crawlers does not conform to real human behavior habits, it will affect the recycling of core indicators such as UVCTR, resulting in misjudgment of effects; at the same time, if crawler data cannot be identified and eliminated, Noise will further affect the accuracy of model training. Data quality issues are the "poison pill" in the "flywheel effect" of data, which will generate positive feedback and continuously amplify defects. We will quantify the above defects and expand the scope of usability in Chapter 4 Calculation Rules.

3.3 Selection of measurement and calculation

Availability can be divided into measurement method and calculation method: measurement is what we often call N nines, and calculation uses mean failure Measured as a function of interval time and average recovery time. In terms of measurement methods, the commonly used quality measurement methods in the industry are shown in Figure 4 below:

Figure 4 Measurement methods

The choice of a few-point scale for the measurement method is not the focus of the quality score at this stage. The N nines used for usability itself are simple enough and comparable, so we focus on the calculation method. Due to the large number of integrated business lines and the recommendation system as a platform product, the relationship between the system and the business is N:N. It is difficult to calculate the availability of the current system for each industry, project and business. A traffic position can belong to the leisure and entertainment business, to the script-killing project, to a part of the core display main path, or to a type of content recommendation. This kind of flexible attribution can be used Requests to aggregate calculations are most appropriate. As shown in Figure 5 below, if availability is a function of requests, it can include not only the quality features we care about in the previous section, but also business-meaningful quality statistics in multiple dimensions.

Figure 5 Measuring quality from the perspective of requests

4 Calculation method

Based on the construction ideas in the previous chapter, from faults to defects, from "yes" or "no" recommendation results to "good" or "bad" recommendation effects, from the whole to each business, we describe a good quality characteristics that points should have. In this chapter, we focus on the calculation logic of indicators, select key defects, define "successful request response", and add the business aggregation dimension of quality scores.

4.1 Calculation formula

Combined with the quality characteristics described in Chapter 3.2, the system quality is evaluated from the perspective of the proportion of successful requests. In actual implementation calculation, it can be divided into the following four Defects at each level:

• System level: The request triggers a system exception, which is a defect response. Common ones include recall timeout, recall failure, recall empty result, etc.
• Data level: If the data used in this request is abnormal, it is a defect response. Common ones include abnormal supply quantity, abnormal label distribution, etc. The actual impact of data on user requests depends on the establishment of data lineage and impact assessment.
• Algorithm level: If the features, models, and strategies used in this request during the recall and sorting process are abnormal, it is a defect response. Common ones include delayed model update, missing features, etc., which affect the expression of recommendation effects.
• Business level: The request triggers business suitability or security compliance requirements, and the requests containing the above results are all defective responses. Common operational feedback includes serious Bad Cases such as supply quality and content security.

If a request experiences a defect in any part of the life cycle, the result is defined as a defect response. The specific defect link is the dimension of analysis and drill-down. We select typical problems (Business pain points, high-frequency quality problems) from the quality characteristics in Chapter 3.2 and the four aspects of the above defects for calculation. Take Figure 6 below as an example:

Figure 6 Quality score calculation method

4.2 Business generalization

The business feature of the comprehensive recommendation system is multi-business Lines, industry differences are large, and there are many recommended material locations. This is reflected in quality measurement. We need aggregate analysis at all levels to guide refined operations, as shown in Figure 7 below:

Figure 7 Aggregation analysis of each business level

There are many medium and low-frequency services in the aggregate. At this time, the fluctuation of the ratio is greatly affected by the absolute value of the request. . For these scenarios, some small traffic bits can be aggregated and only monitored at the industry or project level at the minute level.

4.3 Indicator system

As shown in Figure 8 below, we treat a request responded to by the recommendation system as a product delivery behavior. The proportion of these requests that are defect-free is the quality score of the recommendation system. , is the top-level quality output indicator. Based on the life cycle of the request, first-level input indicators can be established to measure the quality status of the core process, such as recall defect rate, sorting defect rate, etc. The first-level indicators can also be further disassembled to obtain the second-level input indicators. For example, when the recall defect rate is relatively high, the recall null rate, recall timeout rate, etc. can be measured. User requests can also be aggregated vertically, horizontally, and in time dimensions based on business to obtain quality scores with business attributes, which will be more targeted and focused.

Figure 8 Quality Index System

This improved set of quality scores is based on requirements Compared with the original availability calculation method, the unit solves its limitations to a certain extent: it is sensitive to defects, can include the impact of data links, and facilitates aggregate analysis of multiple business dimensions.

4.4 Bloodline Expansion

The quality is calculated based on the granularity of the request. In the data application service, the request is only one of the forms of external data output. After completing the basic quality score, the life cycle of the request should be extended to the entire data link, so that the measurement of quality is complete. At this time, we rely on the blood relationship of the data to associate the data table - business system - C-side traffic to build a panoramic quality portrait, as shown in Figure 9 below:

Figure 9 Recommended system data blood relationship

Blood relationship is the interpersonal relationship generated by marriage and childbirth in human society, such as the relationship between parents and children, the relationship between brothers and sisters , and other kinship relationships derived from it. Data can also generate data kinship relationships through fusion and conversion. The blood relationship of data is divided into databases, data tables, and fields at different levels, which are generally used for data assets (citation popularity calculation, understanding data context), data development (impact analysis, attribution analysis ), data governance (link status tracking, data warehouse management), and data security (security compliance inspection, label propagation). Under the current idea of ​​​​recommending system quality scores, impact analysis is mainly used to expand quality scores, and all requests from faulty nodes are marked and the corresponding scores are deducted.

Under the business semantics of the recommendation system, we define six types of business metadata: snapshot, scheme, component, index, model, feature. Based on the metadata, we build lineage, which can be divided into Provide task access, bloodline analysis, and data export. Task access is divided into a collection module and a warehousing module. When the task access is completed, the nodes and their relationships will be stored in the graph database, and the graph algorithm will be used to establish blood relationships. After the blood relationship is established, the abnormality of the node itself supports system discovery and manual marking, and the impact analysis can be completed automatically. When an abnormality occurs on a node, a message is notified, and the abnormal information will be propagated along the bloodline, which will then affect the quality score calculation of downstream links.

When the exception affects the user end, we try to re-describe the loss in business language. Based on the comprehensive revenue model, the value of each intention UV of each business line can be calculated (users’ visits to merchant details pages and group order details pages are called intention visits), and then use this traffic to generate year-on-year results Visit status and automatically deduce business losses.

5 Indicator Operation

5.1 System Implementation

The system implementation method of quality score relies on burying points and diagnosis. It is recommended that the full link include parameter input, recall pre-processing, recall, recall post-processing, rough sorting, fine sorting, rearrangement, etc. Each link may fail, so data collection needs to cover runtime exceptions. , key input and output information of each link, etc. As shown in Figure 10 below, we asynchronously collect hidden data through Kafka, and then perform data processing according to scenarios: In the production environment, near-real-time ES builds an index to provide fast query services for the past 4 days, and logs 4 days ago are archived in Hive. In addition, The buried point data is parsed through the Flink engine. After necessary diagnosis, scores are calculated in real time and alarm information is pushed. In the test environment, the logs are sorted to MySQL in real time to facilitate testing and troubleshooting. Finally, a structured display of the quality of recommendations at different stages improves the readability of the results.

Figure 10 System implementation of quality score

The improvement of the score system needs to be gradually promoted. For recommendations System, failure to recommend results is the most serious quality problem. The first thing we collect and calculate is the recommendation null result, which corresponds to the result defect rate and recall defect rate in the first-level indicators and the result null rate and recall null rate in the second-level indicators. At the same time, due to the business characteristics of the comprehensive business, there are many industries, and the supply is unevenly distributed in time and space, a large number of cross-cutting filtering conditions will also have empty results, affecting the calculation of quality scores.

How to eliminate empty results that meet business expectations and eliminate quality noise, on the basis of realizing hidden points, diagnosis becomes very important. Taking empty results as an example, we mainly identify them from three aspects: parameter diagnosis, data diagnosis, and link diagnosis. Data diagnosis refers to when an online filtering condition produces an empty result, going back to the source to verify the underlying data a second time and querying whether the bottom table data is empty. If there is indeed no relevant supply in the bottom table, the alarm-free rule will be precipitated and the alarm-free validity period will be set. Within a period of time, the current industry in the current city does lack relevant supply, and the empty result will not be included in the quality score calculation.

If there is supply in the bottom table, it means that there is an abnormality in the data processing or service process, resulting in the inability to recall. Then the error link will be determined through link diagnosis and included in the corresponding quality score calculation. . How to establish a rule matching mechanism ( is the rule engine) is the key to the diagnosis engine. There are many choices for rule engines today, such as EasyRule, Drools, Zools, Aviator, etc. According to the above analysis, the diagnostic engine needs to be able to perform rule diagnosis on request parameters, recommended links, and underlying data. Diagnosis of request parameters and recommended links can be performed through memory parameters, while data diagnosis requires information from third-party storage, so some of it must be customized and developed. Considering the maturity and convenience of human tools, the Aviator expression engine is more suitable. In order to fit the content that needs to be diagnosed, the designed expression diagnosis primitive is as follows:

<span style="color: rgb(106, 115, 125); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">//参数诊断-原语表达</span><br><span style="color: rgb(106, 115, 125); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">//是否符合一定参数的诊断原语</span><br><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">global</span>:<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">check</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">aviator</span>[<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">cityId</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">!=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">nil</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">&&</span> <span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">include</span>(<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">string</span>.<span style="color: rgb(0, 92, 197); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">split</span>(<span style="color: rgb(102, 153, 0); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">'1,2,3,4,5,6,7,8,9,10,16,17'</span>,<span style="color: rgb(102, 153, 0); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">','</span>),<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">str</span>(<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">cityId</span>))]<br><br><span style="color: rgb(106, 115, 125); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">//链路诊断-原语表达</span><br><span style="color: rgb(106, 115, 125); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">//1、召回异常诊断原语</span><br><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">global</span>:<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">recallException</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">param</span>[<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">$</span>{<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">recall</span><span style="color: rgb(0, 92, 197); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">#exception</span>#}],<br><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">global</span>:<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">check</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">aviator</span>[<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">recallException</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">!=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">nil</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">&&</span> <span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">recallException</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">!=</span><span style="color: rgb(102, 153, 0); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">''</span> ]<br><span style="color: rgb(106, 115, 125); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">//2、召回空无异常的诊断原语</span><br><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">global</span>:<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">recallEmpty</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">param</span>[<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">$</span>{<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">recall</span><span style="color: rgb(0, 92, 197); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">#after</span>#}],<br><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">global</span>:<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">check</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">aviator</span>[<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">recallEmpty</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">!=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">nil</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">&&</span> <span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">recallEmpty</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">!=</span><span style="color: rgb(102, 153, 0); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">''</span> ]<br><span style="color: rgb(106, 115, 125); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">//3、召回不为空，过滤规则执行后为空的诊断原语</span><br><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">global</span>:<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">recallEmptyCode</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">param</span>[<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">$</span>{<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">recall</span><span style="color: rgb(0, 92, 197); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">#after</span>#}],<br><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">global</span>:<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">predictFiltersEmptyCode</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">param</span>[<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">$</span>{<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">predict</span><span style="color: rgb(0, 92, 197); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">#after</span><span style="color: rgb(0, 92, 197); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">#filters</span>#}],<br><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">global</span>:<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">check</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">aviator</span>[(<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">recallEmptyCode</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">==</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">nil</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">||</span> <span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">recallEmptyCode</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">==</span><span style="color: rgb(102, 153, 0); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">''</span>)<span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">&&</span> <span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">predictFiltersEmptyCode</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">!=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">nil</span>]<br><span style="color: rgb(106, 115, 125); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">//4、执行某一具体过滤规则后，导致无结果的匹配</span><br><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">global</span>:<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">filterEmptyCode</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">param</span>[<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">$</span>{<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">PredictStage</span><span style="color: rgb(0, 92, 197); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">#filter</span><span style="color: rgb(0, 92, 197); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">#after</span><span style="color: rgb(0, 92, 197); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">#_compSkRef</span>#}],<br><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">global</span>:<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">check</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">aviator</span>[<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">filterEmptyCode</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">!=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">nil</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">&&</span> <span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">filterEmptyCode</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">==</span><span style="color: rgb(102, 153, 0); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">'deleteItemByConditionalFilter'</span> ]<br><br><span style="color: rgb(106, 115, 125); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">//数据诊断-原语表达（判断底层是否有数据，若没有则为true，否则为false）</span><br><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">global</span>:<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">keys</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">keySpread</span>[<span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">@</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">prefix</span> <span style="color: rgb(0, 92, 197); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">138_</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">ymtags_</span>][<span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">@</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">crossOrder</span> <span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">city_$</span>{<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">cityId</span>}<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">_platform_$</span>{<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">platformNo</span>}<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">_surgery_prj_$</span>{<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">genericLvlIds</span>}],<br><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">global</span>:<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">cnt</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">cellar</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">@</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">cellar</span>[<span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">@</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">count</span> <span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">$</span>{<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">keys</span>}],<br><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">global</span>:<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">check</span><span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">aviator</span>[<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">cnt</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">!=</span><span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">nil</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">&&</span> <span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">cnt</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">!=</span><span style="color: rgb(102, 153, 0); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">''</span> <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">&&</span> <span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">long</span>(<span style="color: rgb(89, 89, 89); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">cnt</span>) <span style="color: rgb(215, 58, 73); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);"> <span style="color: rgb(0, 92, 197); margin: 0px; padding: 0px; background: none 0% 0% / auto repeat scroll padding-box border-box rgba(0, 0, 0, 0);">0</span> ]</span>

5.2 Alarm follow-up

Quality score can be used for real-time monitoring and operational review , team members need to follow up on changes in a timely manner. Generally, alarm systems common to companies configure alarm recipients based on service name granularity. Platform services such as recommendation systems provide services through a unified interface, but the model strategies are maintained by different students, and there is a certain threshold of industry knowledge and understanding between businesses. The default broadcast type of alarms can easily cause alarm storms. Everyone cannot focus on the problems of their own modules, and sometimes miss alarms. For the sake of follow-up rate (As shown in Figure 11 below), we developed a follow-up function based on the existing alarms, routing the alarms of specific traffic levels to the dedicated person in charge, and recording the follow-up Status flow facilitates timely notification and subsequent review. In terms of operations, we build a quality dashboard through data reports and regularly review the quality fluctuations of different businesses.

Figure 11 Alarm follow-up process

5.3 Governance effect

Quality score The implementation starts with the result null rate. It disassembles and collects the recall null rate, model prediction null rate, and rearrangement operator null rate according to the process, and aggregates them into platforms, businesses, forms, projects, and traffic positions by business. Multiple dimensions. Governance actions and results are divided into the following aspects:

• Through tracking and diagnosis, we can determine whether the current empty results are supply problems or quality problems, and exclude 98% of the empty results from being included in the quality score calculation to avoid false alarms. The average daily number of empty result alarms has been reduced from 40 reduced to 5.
• Based on analyzing the null rate of each link in the link process, take governance measures, including data specifications (data layering standardization, labeling specifications), Service architecture (Business isolation, dual media for underlying data, downgrade), change specifications (Configuration of online pipeline inspection, traffic playback) keep the system discovery rate of empty results above 60%.
• Customized development of alarm routing to avoid alarm broadcasting and support marking follow-up status. The empty result alarm follow-up rate ranges from unstatisticable to 100% follow-up for core traffic.

After the management and identification of null results, the current core traffic null rate is 0.01%, which means that 99.99% of core traffic requests are guaranteed to have results. In the construction quality analysis At the same time, it ensures the system discovery rate and alarm follow-up rate.

5.4 Asset Precipitation

The recommendation system delivers the value of data. Only when data is capitalized can this value be sustainable and add value. The process of building a recommendation system quality model is actually also about data asset accumulation. After data is collected, it becomes an asset and generally must meet the following four conditions: flowable, measurable, controllable, and value-added. These are all covered in the calculation method in Chapter 4. The process of indicator operation is also the process of accumulating quality knowledge assets. How does the software defect model affect the final product delivery quality? Is there any correlation or causality between them? Is this impact explicitly involved in score calculation or is it indirect? During the quality sub-operation process, we can gradually fill in the quality map in our minds and form topological relationships between indicators, defects, and indicators and defects. This is a process of capitalizing quality. For example, through the business practice of recommendation systems, we found that 80% of online failures are caused by release, and 80% of release failures are caused by data release. This can guide us to reduce online failures through governance of data release.

6 Future Plan

Based on usability, we adjusted the calculation method and established a multi-level recommendation system quality score, and expanded it to various recommended materials and various The core of the business module is that we have completed the cognitive iteration from the "availability" of external services to the "good or bad" of external services, which is also the basis for quality and refined operations. The follow-up plan is, on the one hand, to continue to enrich the calculation and link coverage of the quality model; on the other hand, we will do more quality governance work based on the quality model. Some of the directions that we will focus on and iterate in the future include:

• By improving burying points and diagnosis, we will gradually implement indicators at all levels in the quality sub-system, enrich the connotation of quality sub-systems, and accommodate more quality problems.
• By building multi-level recommendation flexible downgrades, iterate on the understanding of quality scores and quantify the impact of different downgrades on the system.
• Optimize the accuracy, coverage and timeliness of data lineage, and more accurately and quickly assess the impact of quality issues in a certain link.

7 The authors of this article

Yong Hao, Gengen, Wang Xin, He He, Li Cong, etc. are all from Meituan In-store platform technology department/in-store comprehensive business data team.

The above is the detailed content of Quality model and practice of Meituan’s comprehensive business recommendation system. For more information, please follow other related articles on the PHP Chinese website!