Development history of Internet financial architecture from zero to tens of billions

Looking back, it has been almost three years since the first line of code was typed out from the company’s establishment. The platform’s technical architecture and technical system have also undergone four major upgrades and transformations (the fourth generation architecture system is currently in progress). ), as the end of the year approaches, I also want to take time to review the technological changes behind a small company from zero transactions at the beginning to a transaction volume of over 10 billion.

General introduction

In the Internet financial industry, which has more than 10 billion yuan, it is not actually a big platform, that is, it is a second-tier camp. In fact, every architecture upgrade is accompanied by a major business advancement. The problems encountered in the previous generation system architecture, Some excellent development cases have been accumulated during the business development process, and architecture upgrades will be vigorously promoted in the development of next-generation systems. On the one hand, it can smooth the transition, on the other hand, the company resources can provide strong support, and at the same time, technical partners can use cutting-edge technology, and have a greater sense of accomplishment in development. In this way, we can upgrade the system architecture once in about 9 months. Into our current structure.

Many netizens often ask, what is the TPS of your platform, what is the maximum concurrency, and how is the performance? To be honest, we are a small company, and the most exaggerated is that tens of thousands of people are competing for bids at the same time, but as a medium-sized Internet There are really a lot of things that the financial platform has to do, and it is far more than just these parameters that can be clearly stated; we are not a high-end platform, and the technology we use is also a relatively mainstream open source product at present, but in the process of the company's continuous development I have also encountered a lot of problems, and I have tried my best to use more mainstream, open source solutions that are suitable for us to build the entire system. I would like to share the changes in technology upgrades behind the development of the platform here. I also hope to have more exchanges with you and learn more. Make some suggestions.

We have made four major architectural changes, and each generation of architecture is summarized in one sentence:

• Features of the first generation architecture: business is relatively concentrated, functions meet investment and financial management needs, and can be launched quickly

• Characteristics of the second-generation architecture: distributed system transformation, platformization beginning to take shape, various vertical business systems being built and launched, product-side greatly enriching user investment, and big data platform research and use

• Features of the third generation architecture; SOA governance, using zookeeper as the registration center, dubbo as the monitoring and dispatching center; cas to achieve single sign-on, using shiro for permission control

• Characteristics of the fourth generation architecture: fully enable the microservice development model, with springboot+springcloud technology as the technical support for the fourth generation architecture

Detailed introduction below

First generation system architecture

2014 should be regarded as the first year of Internet finance. In fact, many Internet companies have used various models to survive before, and they have been tepid. However, in 2014, they suddenly became popular. The first was Pingdaizhijia and Pingdao Tianyan. This kind of third-party website traffic suddenly increased, and then media reports continued to follow up. Later, there were more and more reports about various Internet financial companies receiving XXX US dollars of investment, and the policies gradually became clear, so many large companies (groups) ) also took advantage of this craze to follow up, including us.

The main purpose of the first-generation system was to grab time. The company hoped to ensure that the system was online in the shortest possible time. At that time, the mobile wave had already started, so it was decided to prioritize the launch of the mobile version, and the website could not be considered for the time being. At that time, the company had technical reserves of two development languages, PHP and Java. Because PHP had great advantages in rapid development, it decided to adopt the model of front-end PHP + back-end Java. The system is divided into three layers: User layer: Android and IOS mobile terminals; Interface layer: PHP provides user and transaction interfaces; Backend: The backend has two parts, the background and the timing system. The backend uses PHP development and interface layer to share one system, and the other is a timing system, which is responsible for timing tasks such as interest calculation, dividend distribution, expiration, etc., and is developed using Java.

For basic services and middleware, mysql provides the most basic master-slave support. The first-generation system only uses the main library of mysql, and the slave library is only a synchronous backup; memcached is used to handle the concurrency problem of user bidding, and only this part is used. ;ActiveMQ is used to use secondary market transfer matching and other asynchronous message notifications. Project deployment: PHP is deployed using apache, the timing service uses tomcat6 as the application server, and lvs is used as the front-end apache load. Basically, these technologies are the first generation. The following is the architecture diagram of the first generation system.

After the first-generation system was launched, the construction of websites and H5 (mobile browser or WeChat) systems became particularly prominent. As an Internet financial company, it couldn't bear not having an official website, so it began to develop websites and H5 systems non-stop. In this During this period, the backend that PHP had previously done was taken out and re-planned using Java. So far, PHP is responsible for the three systems of website, APP interface, and H5. A core transaction shared by the three systems, Java is responsible for the backend. For management and timing services, we generally call this architecture the 1.1 generation architecture.

Generation 1.1 system architecture diagram, the green part is the changed part

The shortcomings of the first-generation system are that the business is too concentrated, it is launched in a hurry, and there are many problems in the later period.

Second generation system architecture

The background of the second-generation system is that with the rapid development of the company's business volume, many technical debts owed in the early stage have exploded, and many problems have appeared online. The most serious one was repeated dividend payments to individual users, and they were scolded in various ways. The memory is still fresh now. On the other hand, various business departments have constant demands, and the company's products are in constant demand, so at this stage, they are busy fixing various production problems, while also developing vertical business systems. I was almost driven crazy during that time. The first-generation system was developed in a closed manner. I still haven’t recovered from my stress when I came back, but I rushed to put it on the market again. It was really painful and happy.

The first vertical subsystem to go online was the contract system. At that time, there was no contract after users submitted bids. Many users were very worried and put the priority up. Later, the contract system alone was changed to three versions. The first version only generated pdf, the second phase was online with electronic signatures, the third phase added watermarks, and customized and dynamically generated pdfs; followed by the development of a points system: user invitation, Investment and other production points can be used to exchange for cash coupons, etc.; extract the message system: site messages, text messages, emails, etc.; launch the online monitoring system, business monitoring and service monitoring, and business failure early warning; each business department will continue to raise demands and go online Financial system: financial staff calculates amounts; risk control system: monitors abnormal users and abnormal transactions; develops a sales system for sales; and develops an external access system because it interfaces with many third-party systems.

The first-generation system was built in a hurry, and the product interface was terrible. Then we started planning website 2.0, APP2.0, and H52.0. In response to the needs of the front-end system, we developed a CMS system on the back-end to publish project and company announcements, news, etc. ; The second-generation product side generally plans a lot of big data analysis needs. It will display on the official website where investment preferences and investment amounts go after full data analysis. The front-end uses a map to display it, and there will also be repayments for individuals. Calendar, collected data analysis, etc., because they need to run a full amount of data, they are designed to be processed offline during planning. The data is synchronized from the mysql slave library to the mongodb cluster, and mongdo's mapreduce technology is used to process large amounts of data. Our database layer becomes the following architecture

Mysql is synchronized to mongodb in real time. We use the tool tungsten-relicator, which will start a monitoring agent on the mysql server side to monitor the binlog log of mysql in real time. At the same time, a service is also started on the mongodb server side. On the client side, the agent monitors the data changes and sends them to the server. The server parses and inserts them into the mongodb cluster to achieve real-time synchronization, as shown in the picture above. I originally wrote an article to introduce it: Big Data Practice-Data Synchronization Article tungsten-relicator (mysql->mongo), in fact, this tool is not particularly stable in use, but there were not many options at the beginning. Fortunately, it became stable after I gradually became familiar with it.

We boldly used golang to develop the data cleaning system. The version of golang we used at that time was 1.3, but now it is 1.8. I had never been exposed to it before and it trained the team. Fortunately, the golang language itself is very simple and efficient. Although I stepped on the N It was a lot of pitfalls, but in the end we put it into production on time; later we used golang to develop a backend, which was based on the beego framework. The big data analysis system was later upgraded to another generation. In each front-end business system, the UI user layer made a lot of efforts to collect user data. It received it through activeMQ transmission and finally stored it in mongodb. After cleaning the data, the cleaned results were stored. into the result database for use by the front-end business system; later, a new version of the data analysis system was built using beego+echart.

Architecture diagram of big data system

Because the pressure on the back-end database continues to increase, the back-end management system and business system have been separated from master and slave; the back-end management system has added cache and started redis for caching; an independent image server has been built using nginx; second-generation system development During the process, it was also the fastest growing stage of the company, with numerous activities launched online.

Second generation system architecture diagram:

Let’s summarize in a moment:
The second-generation architecture has launched various business systems, separated master and slave, and built a big data platform to provide a technical foundation for more data processing in the future
Disadvantages: After each business system is divided, the calls between each project are complicated; there are many back-end systems, and there are separate account systems between each system. Operations need to switch back and forth to complete platform operation monitoring

third generation system architecture

After the development of the second-generation system was completed, we were left with three painful problems. The first was that as business systems continued to increase, the calling relationships between systems increased exponentially. In the early days of the third-generation system, we developed Many basic components have been added, exacerbating this problem; the second problem is complementary to the first problem, and there are too many calling relationships between systems. If you move one of the subsystems, you may need to modify the configuration file of the associated system and restart the service. , often because of updating one system, other systems also need to be passively updated, which makes it complicated to put into production and switch when problems arise; the third problem is that we have developed many back-end systems, but the accounts are not unified. Each subsystem has its own account center, and operations Business staff need to log in back and forth to complete their daily work. As the business volume increases, this problem has become increasingly prominent.

So we started research and system selection, etc. The first problem to solve was to introduce SOA service governance and solve the decoupling between systems through service registration and discovery. We investigated a lot at that time and finally selected dubbo for no other reason than A large number of people walked through the water using the basic water. To solve the second problem, we introduced the configuration center. At that time, we researched 360’s Qihoo360/QConf, Spring’s spring-cloud-config, Taobao’s diamond, and Baidu’s disconf. Finally, after struggling for a long time, we chose disconf, which was perfect for spring cloud. We passed it by, but it was from here that we noticed that spring-cloud and Spring-boot laid the foundation for the fourth generation architecture selection. In fact, disconf was only used in a small number of projects and was not fully promoted; To solve the third problem is the account center, which uses cas to implement single sign-on, shiro for permission control, and dubbo to provide server interfaces such as the permission list after login.

Architecture diagram after transformation

On this basis, we extracted many basic components. The comomn component handles common basic classes, including character classes, date classes, encryption classes..., built a fastDFS cluster to process the file system, and built a redis cluster. Testing; independently developed a scheduled scheduling system, integrating all scheduled tasks into the scheduling system. Whichever system needs scheduled tasks can automatically add scheduling strategies to the page; the front-end PHP has made system transformations, master-slave separation, static optimization, etc.

Later, the company started the construction of a crowdfunding platform. This time the system was completely developed in Java language, and the app side adopted a hybrid development model. All first-level pages of the APP were developed natively, and all second-level pages were H5+vue. In this model, all backends use dubbo for service. The final architecture is as follows:

In the figure, only part of the system is listed, and other services are used instead.

The third generation system launched SOA service governance and introduced a unified account center and basic components; the disadvantage is that the development environment is more complex

Fourth generation system architecture

People are always dissatisfied, and technology always hopes to use the best architecture system. During the development of the third-generation system architecture, I learned about spring cloud and spring boot. After continuous learning, I became more and more aware of the convenience of spring boot. I really like the advantages of rapid development. The spring cloud system also fully meets all aspects that need to be considered in a large system. The concept of microservices is constantly being proposed. The above is the technical background; on the other hand, the country has begun to strictly require P2P companies to have deposits with banks. After analyzing the relevant interfaces of the bank, we found that if we strictly follow the rules, our system needs major transformation. At the same time, in order to meet regulatory requirements, the company has also developed Baitiao related products, which is also a big project. Taking advantage of the above two backgrounds, we Decided to fully embrace microservices while working on bank depository and Baitiao projects.

As for why we abandon dubbo and fully embrace spring cloud, there are three reasons. 1. Dubbo has not been updated for many years, and spring cloud is constantly being updated and upgraded. 2. Dubbo mainly does service management and monitoring, and spring cloud almost considers all microservices. It requires all aspects, such as unified configuration center and routing center; 3. Spring cloud is non-intrusive and perfectly integrated with other spring projects, making development more efficient.

Now that we have chosen to use spring boot + spring cloud for transformation, the selection of microservice technology has been decided. So how to start the transformation? After all, the new generation system transformation must not affect the original business at the same time. The most important thing is The problem is that although the original system was developed according to the distributed development model, some systems still share a database due to the old system. Microservices require each independent subsystem to have its own independent library operation. Others If the system needs to modify or query subsystem data, it needs to obtain it through inter-service interface calls. Therefore, we plan to start the springcloud project from newly developed projects and projects that need to be transformed. Other systems will temporarily communicate through router mode. The final system architecture diagram is as follows:

There is no end point on the road of architecture. Change is eternal, and the upgrade of the architecture is to better support the business. The two complement each other.