To solve the difficulty of reproducing large models and collaborating, this post-95s student team created a domestic AI open source community

In the past decade, AI technology has experienced tremendous leaps. Whether in natural language processing, image recognition, or more innovative fields, the impact of AI has been ubiquitous.

With the explosive growth in the number of research, academia and industry are also facing some challenges, including the problems of "paper reproduction" and "interdisciplinary collaboration". Especially when we come to the era of large models, facing model research with tens of billions of parameters, open source, reproduction, and collaboration become more important, but the difficulty becomes even higher.

Paper reproduction is first of all an important reference factor for judging the value of a result. At the same time, in the rapidly developing field of AI, ensuring the reproducibility of research will better promote the accumulation of knowledge and the popularization of technology. It is also the key to maintaining academic integrity and promoting continuous innovation. Faced with these problems, it is particularly important to advocate open science and transparent research. By open source code, data and experimental details, providing a lower-cost computing platform for replication needs, and providing interactive programs to support rapid replication, we may be able to build a more robust and efficient way to advance scientific research. A solid foundation.

If we talk about the problem of "difficulty to reproduce", it is like adding a high wall to the "dialogue" between researchers. The problem of "difficulty in collaboration" also creates an invisible barrier for interdisciplinary cooperation.

In the era of large models, how to build a convenient platform that can lower the threshold for communication and collaboration has become a major challenge. The traditional software development collaboration methods we are familiar with, such as Git-based code management and version control, may no longer be applicable in scenarios such as AI R&D that rely more on experiments than deterministic processes. Its complex experimental version management and relatively complex High usage and deployment thresholds often hinder communication and collaboration between experts in different fields. The current AI field requires new collaboration models and tools, including more intuitive and easy-to-use version control and collaboration platforms, so that experts with non-technical backgrounds can easily participate in the model development, evaluation and demonstration process.

In other words, both scientific researchers and practitioners hope to achieve more efficient and in-depth collaboration based on the sharing of knowledge and technology, and promote further development in the field of AI.

In this context, a new AI open source community platform "SwanHub" was born.

Experience address: https://swanhub.co/

It is worth noting that SwanHub has a very good team from Xi'an University of Electronic Science and Technology. A team of young graduate students and undergraduate students. The four members of the team are all born after 1995. They not only have rich experience in AI research, but also have a passion for open source. Under the leadership of their instructors Professor Wu Jiaji and Professor Tan Mingzhou, they built a one-stop collaborative development, open source sharing, and visual display platform for AI models from scratch, aiming to solve the current difficulties in reproducing, deploying, and managing AI models. difficult core question.

What problems does SwanHub solve?

In the SwanHub platform, AI researchers and practitioners not only have access to rich open source model and tool resources, but also enjoy the convenience and efficiency of collaborative development. Currently, SwanHub has launched several core functions, including AI model hosting, visual display, etc.

First of all, AI researchers can host their code on SwanHub for version management, just like using the Github hosting platform. However, compared with GitHub, SwanHub supports higher-capacity large file hosting, and researchers can host and version manage model weight files of up to dozens of GB.

"Visual display" is also a highlight of SwanHub. Many research articles published in top conferences and journals will demonstrate beautiful effects and innovative technologies. But in practice, engineers and researchers trying to replicate these findings often run into difficulties. For this reason, in recent years, many academic conferences have required authors to submit not only code, but also a certain number of demos to provide more sufficient research information, and SwanHub provides just such a platform for publicly displaying results and obtaining more scientific research traffic.

As shown in the figure below, the SwanHub platform provides a simple model Demo deployment workflow, allowing users to easily deploy code and model weights into a visual and interactive model by clicking a few buttons. The webpage Demo enables real-time online testing of the inference results of the AI ​​model, and supports sharing with collaborators, reviewers, peers, or making it public on the Internet. SwanHub also supports mainstream visualization frameworks such as Gradio, StreamLit, etc.

SwanHub not only provides a platform for researchers who want to share their results to display their open source models, other researchers can also easily Access other researchers’ open source results, experience demos, and conduct academic discussions in the discussion forum.

In addition, the team said that SwanHub will soon launch two functions: the first function is a one-stop code deployment service, allowing users to easily deploy machine learning models as a cloud service API (Application Programming Interface) ), and also supports being open to the whole community or private use. For researchers who open the API of their own models, their research results can be more quickly integrated into various application scenarios and the impact of their research can be improved; at the same time, for users, their research results can also be made available more quickly. The application has access to various powerful open source AI capabilities. The second function is the model experiment management tool SwanLab. Users can connect their own experiment logging program to SwanLab to realize online experiment log recording and management functions. The platform can not only help record training logs and host intermediate models, but also provide visual training results, training completion message push, hyperparameter recording and recommendation, model cross-version comparison and other functions, which facilitates researchers to quickly trial and error and develop, as well as improve many aspects. The efficiency of collaborative training between people.

# The model training log custard of the development of development

##qi Four young people Technical Ideal

What few people know is that behind the SwanHub open source community platform are four young "post-95s" members.

Lin Zeyi, Chen Shaohong, Han Xiangyu, and Lei Qingyang met on the campus of Xi'an University of Electronic Science and Technology. They formed a club called "Lightyear Technology Studio" out of their respective interests in technology. Later, several like-minded young people jointly founded "Ji Chuang Studio" and embarked on the journey of technological entrepreneurship.

The inspiration for building SwanHub not only comes from the team’s insight into the needs of the open source community based on the current AI field, but it is also related to their own research experience.

In the process of engaging in AI research, they often feel the needs and challenges from internal collaboration and project presentation. Although some commonly used open source platforms can provide basic hosting functions, they often lack a model-based visual collaboration section, making it difficult to unite efforts among laboratory members.

For most researchers, questions like this are very common. On the one hand, the difficulties caused by complex experimental versions and multi-person collaboration will limit the development of research projects. On the other hand, the difficulty of deploying the model and the difficulty of reproducing the training process also hinder the team's internal communication and knowledge accumulation. In daily academic exchange activities, they also lack a platform to intuitively display their results and accumulation.

“These factors have further strengthened our desire to build a collaboration and display platform of our own.” Lin Zeyi, head of the SwanHub project, said. "We hope to provide an open source community for the field of AI to help more scientific researchers and practitioners in aspects such as paper reproduction, technology selection, and technology sharing. In addition, we hope that this model-based visible and interactive The collaboration model can improve the iteration speed of AI projects and the efficiency of communication between team members, reducing unnecessary waiting and communication costs."

Therefore, the original idea of ​​the SwanHub project was to provide a complete set of AI workflows , from papers to open source code to deployment and visual presentation. Through this set of workflows, people can easily build visual demonstrations while conducting experiments and submitting papers for rapid reproduction and testing by peers. On the other hand, academic projects that provide interactive demos are more likely to gain higher dissemination and better reputation, thereby gaining higher academic influence.

At the same time, the team also considered the needs of developers in the industry. Different from traditional software development, AI development is an experimental science. Especially in the era of large models, the development and testing ideas of technology have also undergone great changes: in practical applications, although the model may not perform well on some objective indicators, They perform well, but how to use their "intelligence" in actual scenarios and how to integrate them into professional workflows often requires experts in related fields to conduct in-depth actual effect testing, and this process is also full of challenges.

Regarding this point, Chen Shaohong, a member of the SwanHub project, has a profound experience. His AI research team once participated in a project to develop video processing algorithms for a smartphone manufacturer. At that time, the research team members were scattered in multiple cities across the country, and most of the work required online collaboration. However, judging from the entire online process of algorithm update, verification, client-side deployment, and feedback, each model iteration takes "1.5 weeks" as a unit, which is obviously not able to keep up with the originally planned implementation rhythm of the project.

In order to speed up the algorithm update efficiency, Chen Shaohong recommended that the research team use SwanHub. After training a version of the model, the demo can be quickly updated on the platform. Personnel from various departments of the manufacturer, including PM, product manager, tester , markets, and other researchers can test the effects online and provide diverse improvement feedback, which greatly improves the communication and collaboration efficiency between the two parties, and also greatly increases the iteration speed of the model.

This kind of interdisciplinary collaboration was difficult to achieve in previous code-centered collaborations - for example, it was a hindrance to ask a project member from the marketing department to install the environment and run the project. This is a huge thing, and the platform with Demo as the core makes cross-field collaboration possible.

The value of open source: the driving force of AI technological change

Twenty years ago, a book called "Open Source: Voice of the Revolution" It once took the tech world by storm. This book deeply explores and records the understanding and elaboration of open source culture by more than a dozen open source pioneers, including legendary figures such as Linus Torvalds, the father of Linux, and Richard Stallman, the founder of the free software movement.

For example, Linus Torvalds has always been a loyal supporter of open source and once publicly expressed: "The future is open source everything." For more than 30 years, he has spared no effort to The Linux community has put their efforts into making Linux start as a free operating system with only a few hundred users, and gradually grown into a great and creative community.

SwanHub team members’ enthusiasm for open source originated from this book. They even tried to translate the manuscript, which had only the original English version, into Chinese. The translation process gave them a deeper understanding of the role of open source in academic exchanges and It plays a huge role in promoting the development of science and technology.

This is also the deep value of building the SwanHub open source community. Looking at the field of AI today, most of the impactful advances are deeply rooted in the principles of open science and open source. These principles not only advocate the free dissemination and sharing of knowledge, but also substantially promote scientific research collaboration and innovation on a global scale.

The "Transformers Library" of the Hugging Face community is a classic example: this company, founded in 2016, quickly gained wide recognition from the AI ​​community with its easy-to-use interface and large number of pre-trained models. and a warm welcome. It not only provides a platform for publishing, sharing and collaboration, but also opens up an innovative collaboration model, which greatly lowers the threshold for using deep learning models, allowing more developers and researchers to apply these models. into actual projects and research.

Most importantly, the Hugging Face community encourages and facilitates collaboration on a global scale. Developers and researchers share their own developed models, contribute code, ask questions, and work together to find solutions on this platform. This collaborative approach to brainstorming has greatly promoted the development of AI technology and has also opened up some cutting-edge technologies that may have been closed research to the public.

The success of Hugging Face is not accidental. It reveals that an open and collaborative technology community has played a significant role in promoting technological progress. When scientific researchers have channels to openly share research results such as data, methodologies, models, and tools, their contributions can become the common wealth of the research community.

This practice of open sharing allows other researchers to stand on the shoulders of "giants" and not only see further, but also continue to explore and innovate. In such an environment, the development of AI technology can rapidly advance in a positive cycle.

At this point, the goals of SwanHub and Hugging Face are basically the same. Regarding the future of SwanHub, the team hopes to continue to improve SwanHub's capabilities and experience in collaboration, deployment, community, etc., and will build a tool matrix around SwanHub, including large model modular programming tool SwanChain, model experiment management tool SwanLab, etc., covering AI the entire life cycle of research and continue to take the open source path.

Nowadays, big models are booming and new results are emerging. Companies like Google and OpenAI may not have a "moat", but the power of open source is constantly rising. and catching up. Among the factors that formed this situation were the joint efforts of countless open source advocates.

The above is the detailed content of To solve the difficulty of reproducing large models and collaborating, this post-95s student team created a domestic AI open source community. For more information, please follow other related articles on the PHP Chinese website!