Seven industry insiders discuss how to measure the business impact of AI

AI is evolving and transforming, both as a technology itself and regarding how this technology is used. More and more companies are taking AI testing projects out of the laboratory and deploying them on a large scale, and some of them have achieved significant benefits. Regardless of the uncertainty surrounding AI, ignoring its potential puts companies that are still running their business in old ways at potential risk.

However, for many enterprise organizations, obtaining value from AI may be elusive. Their models may not be well-tuned, their training data sets may not be large enough, customers may be skeptical, and there are concerns about bias, ethics, and transparency. Applying AI to production before it's ready, or expanding an AI strategy to the next stage without properly reviewing it, can be very costly for the business, or worse. The result is that the business develops in an unfavorable direction.

So how do you know whether an AI project will change or subvert your business? Without hard ROI numbers, companies must achieve innovation in a certain way. Let’s take a look at how these IT leaders and industry insiders measure the value of AI:

Mature technology and Breakthrough Technology

Measuring the business value of any initiative or technology is not always a linear calculation process, and AI is certainly no exception, especially when maturity and business potential are taken into consideration. Validated and predictive variables (e.g. data mining, savings in cost and training time, investment, ability to promote new uses) will have an impact on decisions, especially in terms of acceptable ROI, but regardless, the technology It's critical to have a level of trust, whether it's a new or established technology.

For example, at NASA’s Jet Propulsion Laboratory, one of the key factors in measuring the return on investment of AI projects is the maturity of the technology.

Chris Mattmann, chief technology and innovation officer of the laboratory, said that the uses of some AI technologies are already very mature, taking automated business processes as an example.

He said: "Every company has some boring stuff, and we do too. We have automated these processes, such as ticket processing, search, data mining, using AI to review contracts and subcontracts."

The lab uses a number of commercially available technologies to do this, including DataRobot and Google Cloud. To determine whether a particular technology is worth investing in, Mattmann said they consider whether it will save cost, time and resources. "If the technology has matured, it will be reflected."

For those technologies that are still at a medium level of maturity, the laboratory will focus on whether the technology has the ability to open new uses and what the cost is. How many. "If we were to go to Mars, for example, there would be a thin conduit for deep space telecommunications," Mattmann said. Right now, they have enough bandwidth to send about 200 photos a day from Mars to Earth.

"The rover we sent to Mars has a brain about the size of a pea inside, running an iPhone 1 processor. We only put things in space that are resistant to radiation so they can withstand it Deep space environment. We know that the chips that perform well are often older chips, so we will not use advanced artificial intelligence or machine learning on the rover."

However, initially as a NASA's Ingenuity Mars helicopter, which is a technology demonstration rather than the centerpiece of an exploration mission, is powered by Qualcomm's Snapdragon processor, an AI chip. "This proves that it's possible to do more things with AI using newer chips."

This kind of AI will enable many new use cases that are currently unachievable, such as the Ingenuity Mars Helicopter not having to send back 200 photos a day Instead, AI could be used to analyze these images and send a million text messages to Earth saying, for example, there is a dry lake bed in a specific direction, etc. “We can get a lot more information out of text than we can through images today.”

Ultimately, for the most cutting-edge experimental AI technologies, the measure of success will be whether they can be used in new Scientific use, and whether it can be used to write and publish new papers.

He said: "There is a cost to train and build models."

Companies such as Google and Microsoft have ready access to massive amounts of training data, but at NASA's Jet Propulsion Laboratory, data Sets are difficult to obtain and require PhD-level experts to analyze and label.

“At NASA, our cost to train a new AI model is 10 to 20 times higher than in the commercial industry,” Mattmann said.

Here, the emergence of new technologies can allow NASA to build AI models while reducing the workload of manual labeling. For example, generative networks can be used to create synthetic training data, he said. Although it is Deep Fakes, it is used for scientific purposes.

AI measurement and scope of impact

If there is no direct way to measure the business impact of AI projects, companies will instead mine data from relevant key performance indicators, or KPIs. These proxy variables are usually related to business goals and may include customer satisfaction, time to market, employee retention, etc.

Atlantic Health System, an American medical service provider, is a good example. Sunil Dadlani, the company's senior vice president and chief information officer, said patients are at the center of every decision. In many ways, they measure AI's return on investment by observing improvements in patient care. These patient-centered metrics include shorter hospital stays, shorter treatment times, faster insurance eligibility verification, and faster prior insurance authorization, among others, he said.

Another project involving the use of AI is to help radiologists review scans. In this case, the KPI measured is how often the radiologist is alerted to potential anomalies. Dadlani said: "As of April 2022, 99% of our radiologists reported using AI to analyze more than 12,000 studies, triggering nearly 600 alerts. As a result, doctors can address potentially serious issues as quickly as possible."

At RSM, the fifth largest accounting firm in the United States, investment in AI follows two closely linked paths: one is productivity and analytical tools to help employees work better, and the other is similar tools used by clients. said Richard Davis, a partner in the firm's management consulting, business and technology transformation team.

For example, when working with clients, RSM may be asked to pull data from multiple systems (including accounting, sales and marketing, human resources, logistics) and consolidate everything into a single dashboard . AI can help them speed up that process, Davis said, and then AI can be used to see where the workflow goes through those systems and what challenges and roadblocks there might be.

So how do companies know if their AI is heading in the right direction?

“First of all, we can measure the usage of the tool very clearly,” Davis said. He did not provide details on RSM’s investment in AI projects or the return on investment, but he said, “Over time, Over time, what we want to see is engagement being delivered more effectively.”

Increased engagement will improve productivity, he said. "So if it used to take us a week to do something, now our goal might be to get it down to a day."

Focus on business benefits

Measuring the success of AI , can be very subjective. Eugenio Zuccarelli, an AI research scientist at MIT and a retail industry data scientist, said that evaluating an AI project is as much an art as developing AI itself.

Still, it’s important to be able to explain AI’s impact on the business, Zuccarelli said. “KPIs should not be set around the model itself, but around business and people metrics, which should be the ultimate goal of the project.” Otherwise, it’s easy to choose a project that looks successful but doesn’t actually translate into an effective impact on the business. technical indicators.

Zuccarelli, who also held data science roles at BMW and Telstra, warned against measuring the progress of AI projects in isolation. For example, if the goal of an AI project is to improve something that is already being improved for other reasons, a control group is needed to determine how much of the improvement is actually caused by the AI.

Vladislav Shapiro, who has many years of experience in the financial services industry, said that other valuable KPIs for AI projects could be, for example, reducing false alarms or automatically removing excessive privileges. He is also the founder of Costidity, a consulting group specializing in IT security, identity governance and management.

Recently he was responsible for a security deployment driven by AI, which resulted in a threefold reduction in false alarm rates and the automation of many previously manual processes.

He said: "When you show these numbers to C-level executives, they will understand that all of the above measures reduce the risk of data breaches and enhance accountability and governance."

Measuring success step by step

Sanjay Srivastava, chief digital strategy officer of global professional services company Genpact, said that the cost savings achieved through automation are the simplest and clearest way to demonstrate the economic benefits of AI projects. . But at the same time, AI can also promote new revenue streams and even completely change the business model of enterprises.

For example, one aircraft engine manufacturer discovered that they could start servicing engines by using AI to better predict failures and improve logistics. "For the end consumer, purchasing flight miles is better than purchasing the engine itself. This is a new business model that changes the way companies operate because of the empowerment of AI."

Moreover, The business impact is also evident.

So, to justify investing in AI during that time, the manufacturer needs to establish a long-term goal and then translate that goal into several short-term projects that can be measured in other ways.

He said: "Instead of saying, 'In ten years, we will change the industry,' it is better to say, 'In the first year, we start thinking about what parts we need to inventory,' you don't have the ability to disrupt the industry, you Just say, 'We need the right number of parts,' and it's a year-long project with the goal of optimizing your warehouse system and reducing your investment in inventory."

In addition to supply chain optimization , other short-term progress measures include customer satisfaction.

“For example, if a plane is stuck in Mumbai for five days waiting for a certain part, the customer will feel it.”

Alignment with strategic vision

Then there’s the face The reality is that some AI projects may impact profits in the short term, but remain important and transformative in the long term. For example, one business deployed a customer service chatbot that eliminated many tedious tasks. Gartner analyst Whit Andrews said: "But chatbots can also be detrimental, because some salespeople are very good at upselling and want to interact with people, so enterprise organizations may not want all robots."

He said that in the final analysis it depends on what kind of enterprise you want to be. "At some point, you have to ask yourself if you are the kind of business where if a delivery is messed up, the customer calls and asks where the goods are, and then you engage with the customer and try to get them to take delivery once a month instead."

If the organization wants to drive transformation by demonstrating measurable ROI and has a customer-centric vision, it may be possible to skip short-term returns on profits in favor of other metrics that may be more meaningful. superior.

“A fully automated organization may be more successful as their market share gradually increases, but you can develop your data so you can reach the right people at the right time. If there is Anything you can point to and say that, logically, would make our customers happier and our employees more successful, then go ahead and do it."

The above is the detailed content of Seven industry insiders discuss how to measure the business impact of AI. For more information, please follow other related articles on the PHP Chinese website!