Your AI Initiative is Likely to Fail

Three pivotal considerations for AI-based healthcare solutions

Jim Roman, Simon Johnson

November 29, 2023

12 Min Read
Artificial Intelligence
Shutthiphong Chandaeng / iStock via Getty Images

Imagine regaining consciousness just as you’re being wheeled into the ER with excruciating pain radiating from your chest. The doctor tells you that, although your ailment isn’t life-threatening, she believes she knows what’s wrong and can address it with a very expensive procedure. Although you’re semi-lucid, you think to ask a few questions:

“Just how expensive are we talking?” 

The doctor states that the surgery will be extremely expensive, then whispers in your ear that more than half the time the total cost is over 189% of the original estimate. “How can you put a price on your health,” you ask yourself rhetorically. 

“How long will the procedure take?”  

The duration is much longer than you expected, and you’re told that 40-50% of the time it takes even longer. “Fine; better to be thorough,” you think.

Most importantly, “What’s your success rate?” 

After more than a moment of awkward silence, you hear…14%

Do you go forward with the procedure? As an executive tasked with green lighting any software development project, this is your dilemma as these numbers represent the success (or failure) rates for such undertakings.[1] If you add an inconclusive diagnosis to this scenario to make it even more complex, you’ve got an AI initiative. 

Developing any type of software is an extremely complicated endeavor, and AI in a regulated environment takes it to another level. If they’re being honest, many executives don’t even know what AI is or why they (or their customers) need it. Like digital transformation and big data, AI has become the buzzword du jour.  

With this in mind, let’s demystify some of the AI complexity by starting with a few high-level definitions. Then, we’ll highlight the benefits, and ultimately discuss three key factors that must be considered before embarking on the AI path.

Neural Network

In biology, human brain nerve cells (neurons) form a complex, interconnected network, sending and receiving electrical signals to process information.


Artificial intelligence (AI)

A branch of computer science in which systems are developed that can simulate intelligent human behaviors such as learning, reasoning, and problem solving. The core elements of an AI system are:

  • Data: Large datasets are used to train algorithms on how to make predictions.

  • Algorithm: A set of instructions for taking in training data and using the information as a basis for completing tasks.

  • Model: A program that leverages algorithms to analyze datasets, find patterns, and make predictions.

  • Platform: Sets of hardware and software technology on which AI applications are developed and deployed.

  • Integration: The incorporation of AI platforms into existing workflows and systems.

  • Output: Information generated by the model to facilitate decisions within a use case.

Machine learning (ML)

A subset of AI in which systems can learn and evolve over time, albeit with human guidance/retraining, by:

  • Leveraging algorithms with two processing layers (input/output) and statistical models.

  • Storing incoming data, as well as data about actions/decisions it makes.

  • Analyzing that stored data to improve/evolve over time.

Deep learning (DL)

A subset of ML in which algorithms:

  • Contain multiple processing layers that make up an artificial neural network.

  • Receive, process, and analyze a vast amount of unstructured data.

  • Are capable of independent learning, analyzing their own predictions and adjusting their accuracy over time, without human intervention.

  • The benefits of these advancements are more conspicuous than the technical concepts. 

Clinical accuracy

A healthcare provider who can derive actionable information from massive data sets can make better-informed decisions during case planning and in real-time. For example, by leveraging an AI solution, a surgeon can better optimize the margins needed for tissue removal, effectively reducing or eliminating collateral damage to healthy anatomy.

Patient outcomes

In the case of diagnostic imaging, AI’s ability to analyze copious amounts of data with great speed and precision can identify subtle patterns that wouldn’t be evident otherwise. A 2020 study[1] illustrates striking differences between the accuracy rate of tumor classification by radiologists as compared with an AI-enabled system, depending on the type of tumor. 


Adding phenotype data to the equation — a manifestation of a patient’s genetics combined with environmental data — enables predictive analytics which can yield a better diagnosis of the severity of a health issue, at the time of evaluation and in the future. 

Economic value

With value-based care gaining momentum, compensation to healthcare organizations is starting to be determined by patient outcomes rather than the amount of care provided. As such, these organizations are scrutinizing every aspect of their operations to improve their financial health, while concurrently striving to provide the best possible patient care. Higher clinical accuracy and better patient outcomes can prevent unnecessary procedures and reduce readmission rates. The proper application of an AI solution can help streamline workflows, potentially decreasing the staff required to support certain procedures, while increasing the number of cases that can be performed in a day.   

Given these benefits, it’s no surprise that AI applications have gained the most traction in radiology, with cardiology trailing far behind and other areas of medicine “in the dust” (for now). 


Data Source: The Food and Drug Administration | Chart Created by MedAcuity 

Given all the latent potential of AI in healthcare, what’s stopping you from diving right in? In our experience, there are three primary considerations that can impact the success or failure of an AI initiative, each of which is interrelated.  

1. Solve a problem

A solution requires a problem to be solved, which remains a best practice for software development. A market-driven approach, therefore, is the right way to approach any software development project, especially a complex one involving AI. 

Define: Start by identifying an unmet need, which demonstrates an understanding of the market and provides a foundation on which a solution can be built. The benefits of some AI applications are evident, although not all use cases are good candidates for a variety of clinical, practical, or business reasons. Be sure that you can answer these questions:

  • What is the meaningful problem we’re trying to solve and what are the use cases?

  • What attributes are must-have vs nice-to-have?

The answers to these first two questions represent your initial set of requirements to describe what the market needs and serve as inputs to the architecture.

  • What are the options for solving the problem? 

  • Does leveraging AI represent the most appropriate/effective approach?

  • What is the market potential for leveraging an AI solution? 

  • What is the likelihood the AI solution will be accepted by the market/regulatory authorities in which you plan to launch it?

Postulate: Once you’re confident in your answers, generate a hypothesis. Then, create a system architecture and a design based on the requirements as well as a set of well-defined benchmarks and expected outcomes, noting the benefits for each use case. 

Iterate: Develop a proof-of-concept system and test your hypothesis extensively with an appropriate set of participants. Then, “rinse, repeat” making changes based on feedback, much like a formative study, to hone and validate the efficacy of your solution. The following excerpt from the 2020 study referenced earlier illustrates the complexity of this process for an AI initiative:   

  • In this paper, we propose a joint deep learning model where the segmentation can better facilitate the classification of pulmonary GGNs. Based on our observation that masking the nodule to train the model results in better lesion classification, we propose to build a cascade architecture with both segmentation and classification networks. The segmentation model works as a trainable preprocessing module to provide the classification-guided ‘attention’ weight map to the raw CT data to achieve better diagnosis performance. We evaluate our proposed model and compare with other baseline models for 4 clinically significant nodule classification tasks, defined by a combination of pathology types, using 4 classification metrics: Accuracy, Average F1 Score, Matthews Correlation Coefficient (MCC), and Area Under the Receiver Operating Characteristic Curve (AUC). Experimental results show that the proposed method outperforms other baseline models on all the diagnostic classification tasks.

When applied properly, a market-driven approach can prevent organizations from making massive investments in solutions before the right problems are identified and the viability of an AI solution is confirmed.

2. Build on a solid foundation

Looking back on the scenario that opened this article, I presume that most people would opt to receive modern anesthesia over biting on a stick or taking a shot of whiskey to control the pain during such an invasive procedure. Of course, modern pain management techniques and equipment have evolved exponentially in effectiveness and complexity, compared with those used in the 1800s. Similarly, one could argue that the evolution of AI technologies represents a quantum leap in the complexity of medical devices, as the amount of planning and the volume of choices is orders-of-magnitude greater than those required for traditional “unintelligent” systems. 

While the best practices surrounding the generation of solid requirements apply to all software development initiatives, the effort required to develop a robust architecture is exponentially greater in an AI system. 


Because architecture describes what we are building, it is foundational and represents tenets such as reliability, scalability, security, etc. These tenets take on a whole new meaning in AI systems, along with some new ones that require thoughtful consideration:    

Data quality

The output of an AI model is only as good as the training datasets used as inputs. Thus, it is important to define the types of data that will be involved and to create benchmarks against which the accuracy, relevancy, and consistency of the data can be measured. 

Data bias

One of the greatest challenges in confirming the accuracy of data is to ensure that it is not biased due to subconscious preconditioning. If not addressed adequately up front, bias will be perpetuated by the model and expand as the model evolves. In healthcare, this can potentially lead to misdiagnosis, selection of the wrong treatment option, and ultimately, patient harm.   

Model selection

The number of AI model archetypes is vast, with three of the most prominent being supervised (trained by humans), unsupervised (trained by software), and semi-supervised learning (a combination of the two). As such, a decision regarding which (and how many) models to leverage in an AI solution should be based on many factors including the type, volume, and availability of the data, the nature of the algorithms (static vs dynamic), and the complexity of the problem being solved.   


Given their potential for rapid evolution, AI systems must be continuously monitored against performance benchmarks, focusing on the model itself and its impact on other integrated systems.  Without this commitment, a system can be susceptible to many conditions that can negatively impact the accuracy of the learning model, causing it to degrade over time. Two of the most prevalent conditions include:

  • Data drift, which occurs when changes to the input data used for training or making predictions changes, but other aspects of the data model stay the same.

  • Concept drift, which refers to situations in which the relationship between the inputs and outputs of a model changes. This can occur when the context or environment in which the model operates changes, without the model changing.    

Coupled with routine monitoring, periodic recalibration can be a useful tool in overcoming these conditions. 


This concept can be reviewed from multiple angles:

  • Components: Which systems will be incorporated and at what level of interoperability?

  • Deployment: Will the AI solution be distributed to individual devices, to a central cloud-based location, or both?

  • Effectiveness: The acceptability of an AI solution is often tightly coupled with its ability to complement existing use case workflows.


When a medical device is involved in causing patient harm, the FDA’s first reaction is to look at whether the device was “designed for usability,” rather than assuming user-error. As such, when designing an AI solution for healthcare, it’s important to consider how users will interact with it, and to incorporate guardrails to prevent misuse and ensure that the results produced by an evolving system fall within safe, acceptable ranges. Further, there should be a means of checks and balances to enable a human user to validate the information being presented. Finally, in the event of a problem, AI solutions must have a transparent mechanism for tracing the behaviors and results of the system to facilitate root cause analysis.      

3. Don’t skimp

An AI initiative is neither simple or easy. Shortcuts will result in sunk costs, therefore, it’s important to make the right investments to provide the greatest chance of achieving a successful outcome.     


AI initiatives require a different way of thinking and people who think differently. While engineers with deep AI experience are in short supply due to the nascent state of the technology, there is an increased focus on education in this space, and therefore much more knowledge is coming to the market.  While these roles are critical for building, monitoring, and maintaining the system, the contributions from a cross-functional team consisting of data scientists and domain experts are just as impactful in AI solution development, especially in the definition step of the problem-solving process. 


As described in the Iterate step, there is a myriad of options and decisions to be considered, and they are all interrelated. The problem-solving process and development of the architecture cannot be rushed, as short-changing either of these activities up-front will significantly increase the risk of a failed initiative. 


Complexity adds cost. More than half of software development projects exceed their initial budget by 189% or more, and AI adds even more complexity and risk. Once your team provides a budget estimate, tack on an additional 50% (which still may be low) to give you a better projection of what the initiative will realistically cost. If that number is completely untenable, you may want to reconsider the ROI potential of your AI initiative.   

AI has the potential to transform many aspects of healthcare in a positive way, but without the proper forethought or guardrails in place, the results could be tragic. Most people wouldn’t gamble with their lives on a medical procedure with only a 14% success rate, however, taking a hasty approach with an AI initiative has the potential to yield a similar likelihood of failure. In contrast, making the right investments and following best practices related to problem-solving and architecture creation reduce the likelihood of your project flat-lining.    


Jim Roman, VP of business development at MedAcuity

For over a decade with MedAcuity, an engineering firm specializing in medtech software development, Roman has dedicated himself to working with clients to clear the most challenging hurdles and to accelerate the right products to market.

Additional Contributions from:

Simon Johnson, Sr. vice president of sales & marketing at MedAcuity

Johnson leads the go-to-market strategy, sales, and marketing for MedAcuity.

[1] GITNUX Market Data: “The Most Surprising Software Project Failure Statistics and Trends in 2023.”

[2] 3D deep learning-based classification of pulmonary ground glass opacity nodules with automatic segmentation, Wang et al., Journal of Computerized Medical Imaging and Graphics 2020

About the Author(s)

Jim Roman

Vice President of Business Development, MedAcuity

For over a decade with MedAcuity, an engineering firm specializing in medtech software development, Roman has dedicated himself to working with clients to clear the most challenging hurdles and to accelerate the right products to market.

Simon Johnson

Senior Vice President of Sales & Marketing, MedAcuity

Simon Johnson leads the go-to-market strategy, sales, and marketing for MedAcuity.

Sign up for the QMED & MD+DI Daily newsletter.

You May Also Like