Seven IT experts talk: How to start and scale a successful AI pilot project

The U.S. Patent and Trademark Office in Alexandria, Virginia, is using artificial intelligence (AI) projects to speed up the patent classification process, help detect fraud, and expand examiners’ searches for similar patents so that they can classify them at the same time. Search for more documents, and every one of their projects starts with a pilot. "Proofs of concept (PoCs) are a key method we use to understand new technologies, test business value assumptions, reduce risk in project delivery at scale, and inform full production implementation decisions," said Jamie Holcombe, chief information officer at the U.S. Patent and Trademark Office. Once the pilot programs prove successful, the next step is to determine whether to expand them, he said. Then it's time to scale in a real environment and go into full production.

Indian e-commerce provider Flipkart also followed a similar process before deploying a project that helped customers text millions of items in 11 different languages Search and visual search. Now, Flipkart is testing conversational bots, using deep learning to build models that include user intent detection, language translation, speech-to-text and text-to-speech capabilities. Both Flipkart and the U.S. Patent and Trademark Office are rapidly expanding the application of computer vision, natural language processing, machine learning and other artificial intelligence technologies to other aspects of the business.

As exciting as artificial intelligence and machine learning are, there are many initial pilot projects and PoC projects that fail to reach full production. Projects that are already successful need to be part of a strategic plan, have executive support, be able to use the right data, have the required team, have the right technical and business metrics, and project milestones, go through multiple iterations, and be fast. trial. "This process may take a year or two to reach a high-quality level, and you will need to be patient during this period," said Ganapathy Krishnan, vice president of engineering at Flipkart.

Laying the foundation for success

Companies are moving quickly to launch successful AI pilot projects into production and deliver results. Rowan Curran, artificial intelligence, machine learning and data science analyst at Forrester, said: "We have seen that AI projects are entering the mainstream, with 57% of enterprises implementing or expanding their AI projects, and 70% to 75% of enterprises We are seeing clear value from these projects.” In addition, according to a recent Ernst & Young survey, 53% of CIOs and IT leaders said that data and analysis in the field of AI will become the top investment area in the next two years.

But many of these pilot projects are doomed to fail before they even begin, for a number of reasons, starting with a lack of top-down support. "You need executive buy-in, and you have to have the funding," said the USPTO's Holcombe. Some IT leaders believe that launching projects from the middle of the organization or from the bottom up will Reduce the chance of project success. The most successful projects will have support from the CIO and executive commitment to fund the project and integrate AI into the organization's overall digital transformation strategy.

It’s also important to set clear expectations, said Flipkart’s Krishnan. "You should not expect that the projects you deploy will fundamentally change the business. This is a long process and takes time."

PoC can also be an exercise in honing capabilities within the enterprise, which is why An approach taken by the manufacturing company Eli Lilly. “Through the PoC, we experimented and understood the scale dimensions of the technology and project delivery,” said Tim Coleman, vice president of information and digital solutions and chief information officer at Lilly. The team is applying natural language processing capabilities to business areas. Natural language discovery, generation and translation, ranging from clinical and scientific content creation to product development, advanced search and general management functions.

But don’t confuse this capacity-building exercise with pilot projects that need to generate broad transformational value, warns Dan Diasio, global AI consulting leader at EY, “You want to build your capabilities to get there, but in the future When you need to compete with disruptors and have a meaningful impact on investors, you have to take a top-down approach."

This is how healthcare company Atlantic Health System approaches its artificial intelligence and machine learning projects. Atlantic Health System has successfully piloted image evaluation to assist radiologists and pre-authorization automation, which takes imaging orders and schedules them through several process steps. Sunil Dadlani, the company's senior vice president and chief information officer, said: "Artificial intelligence should be part of digital transformation, not an isolated initiative. We have formalized a governance structure and investment plan for artificial intelligence and machine learning." At Eli Lilly, project proposals should pass three criteria before moving forward: they provide business value in terms of return on investment, have an acceptable success rate, and project results must be consistent with business strategy and priorities, Coleman said. Mosaic PV, for example, was one of Eli Lilly's first AI projects focused on adverse drug reaction reporting, with the main driver being to "increase productivity and reduce the cost of dealing with adverse events while maintaining high standards of quality and compliance."

what is the problem?

A successful pilot should start with defining the business problem. “Don’t become a problem seeker through an answer,” says Sanjay Srivastava, chief digital strategist at Genpact, a global professional services firm. Genpact mainly provides consulting services on AI projects to large enterprises. “Projects that focus on business success and start with questions rather than answers tend to do well.”

Then decide whether AI is the best answer. "Is this project suitable for a standard that is complex enough to warrant using AI? If you can do it with a simple, rules-based approach, go for it," Krishnan said. "But if you have hundreds of thousands or even millions of When there are tens of thousands of rules, it is not feasible to use software-based methods."

Do you have the appropriate indicators and data?

Back to the US Patent and Trademark Office, AI projects require two sets of indicators: technical indicators on how to execute the model, and indicators on how to quantify the commercial value of the AI ​​project.

Atlantic Health System ensures success by implementing pilot projects with clear business KPIs for a small part of the business. For example, Atlantic Health System's imaging evaluation system began deployment as a small pilot in neurology and quickly expanded to cardiology and other areas. The team created a successful neurology pilot in eight weeks, demonstrated project results, and gained support from cardiology and all other service lines.

Flipkart, like the USPTO, first focuses on technical model metrics and then runs A/B tests to see what impact it will have on the business. Currently, the team is developing and testing an AI-assisted conversational bot. They started with the metric "answerability," which is the robot's ability to answer questions. Now they are running A/B tests to determine if this will have a measurable impact on the business.

AI projects rely heavily on big data, and you need the speed, volume and variety required, Dadlani said. “If your data quality is not good, you’re not going to see the results you expect.”

Genpact’s Srivastava agrees. "Ninety percent of the work of building an AI system is around data ingestion, coordination, engineering and governance. If you focus on 10% and abandon the 90%, you will fail from the beginning, so you have to build the foundation of data."

You also need to be able to provide continuous feedback between different A/B tests - obtaining data in real time so that you can adjust the model. But your organization may not be able to deliver data quickly and automatically. For example, if you're working on a predictive model and the team doesn't automatically get information about what customers are buying, you can't complete the loop. It’s also important to continue the feedback loop after full deployment, as customer preferences can change over time. If your model doesn't take this into account, you won't get the results you expect, a result known as "model drift."

Will the pilot project be expanded?

Although pilot projects may be fully deployable in your initial expectations, scalability depends on these pilot projects. So, do you have the right resources to scale the pilot project to full deployment. "Instead of having a data repository, hiring a new team, building a data label factory, you may need to simplify your code, introduce new technologies, and push AI and machine learning to the edge," says EY's Diasio. Yes, it’s a whole set of engineering skills.”

Execution Pilot Project

Flipkart adopted cloud and MLOps related capabilities in the pilot project. "Pilot projects require a lot of engineering support from the beginning, they have to iterate frequently, trial and error quickly, and to do that you need MLOps infrastructure from a large cloud service provider." He advised the pilot team They should report regularly on how close they are to meeting their goals, and make sure expectations are set correctly during the pilot.

He said: "If you set 3% early in the pilot, you are doing well." Don't expect to see benefits immediately. Complex pilot projects often struggle to see impact within three months. What you have to do is deploy, find the gap, deploy again, and gradually improve.

Failure during a pilot project does not necessarily mean the end of the pilot project. The USPTO's enhanced classification system failed initially. “From the beginning, we had issues with improperly managed data sets,” Holcombe said. But the team recalibrated and continued the pilot until the system performed significantly better than the manual process. “If you fail, don’t give up. Find out why you failed.” project. At Atlantic Health System, once the initial pilot is completed, it's time to evaluate the results and decide whether to extend the pilot, move forward with production, or reduce losses. Dadlani said: “Pilot projects must provide a perceived measure of success, and only when we see promising results will we say, what will it take to scale, how much time will it take, how long will it take to realize value, the technology infrastructure What investments of resources are required, and how do we implement them."

Eli Lilly's Coleman said there are many reasons why pilot projects fail: insufficient AI skills, insufficient labeled data, unclear project vision or value proposition, lack of Agility, lack of a fast trial-and-error mentality, lack of buy-in from senior executives, and lack of organizational change management to drive business adoption.

You have to report upward the indicators that are critical to financial reporting. For example, if the pricing algorithm predicts $50 million in savings, there may be a gap between what has been achieved to date and what was expected, Diasio: “When you’re talking about large, expensive projects, pilot projects often lack the tools to create that much value. Credibility, so try as hard as possible to document the value that has been achieved."

This is also an opportunity to reassess whether the pilot needs to be scaled up. “A lot of PoCs are very successful technically but don’t make economic sense when it comes to scaling,” Genpact’s Srivastava said. Other considerations include how long it will take to scale the project and what resources are needed.

But when you take the long view, things can change. "Even where scale may not be possible in the short term, a smaller project scope with a high probability of delivery success may still deliver short-term business value, while at the same time the technical capabilities and skills mature to the point where they can overcome the challenges," Coleman said. The extent of the barriers to scale.”

Then there’s infrastructure. You should make sure you check all assumptions when scaling, including configuration, network bandwidth, storage, and compute. "You need a lot of engineering support to scale pilot projects, and that's where cloud-based MLOps infrastructure can help," Krishnan said. Finally, you should make sure you can integrate AI into upstream and downstream workflows. middle. For example, predictive failure capabilities won’t be useful if you don’t integrate them into your upstream supply chain systems to ensure you have the spare parts you need when and where you need them. Likewise, this information can be used downstream to adjust maintenance schedules.

Start slow, fail fast, and wait patiently

The key to a successful AI/ML pilot project is initial planning. You need to get executive support and finance before moving forward. on support. "You have to have executive buy-in," Holcombe said, to involve all stakeholders from the beginning.

AI/ML pilot projects should be conducted as part of an overall digital transformation strategy and have compelling business scenarios, Dadlani said, and achieving the desired results requires patience. Develop technical and business impact metrics that define success. Make sure you have the resources you need, build a team and be prepared to trial and error quickly. Therefore, having the required mix of skills and domain expertise on your team is key to the success of your AI pilot project. "Even in the pilot phase, you need a cross-functional team," he said. "We want to make sure that everyone is involved in the pilot project because it will be part of the actual workflow and they have to be involved from the beginning. ”

Organizations that don’t have all the talent they need should consider building a hybrid team with an external partner, while small and medium-sized companies may need to outsource more of the roles – if they can find the right talent. “Outsourcing is very difficult if you don’t have the right AI/ML engineers and data engineers,” Srivastava says. What’s more, you need to have people on your team who understand both machine learning and the industry (such as manufacturing). ) personnel. This is not an easy skill set to find, so cross-training is crucial.

Finally, you want to consider a target project that can produce actual business results and then expand into other areas of the business, like Atlantic Health System did with its machine learning-based imaging assessment system project.

Once the pilot moves into full production, build on the work you’ve already done. Keep business units informed on the progress of the pilot, demonstrate what the project will do when fully deployed, and develop a platform on which other business units can use their own applications. Srivastava said: “Today, the pace of change is the slowest it has ever been, and companies that want to disrupt and grow must change the way they drive value, and you can’t do that without AI. If you don’t invest If you use artificial intelligence, you will appear to be helpless."

The above is the detailed content of Seven IT experts talk: How to start and scale a successful AI pilot project. For more information, please follow other related articles on the PHP Chinese website!