Generative AI is rapidly changing healthcare with solutions bringing efficiency to clinical documentation, decision support, and patient communication, to name just a few applications. These new solution tools help reduce physician administrative burden (“pajama time”), accelerate clinical decisions, and improve patient engagement. An investigation by Menlo Park found that 22% of healthcare organizations have implemented AI tools.
However, as with any major technological advancement, there are challenges. The main concern is not only how GenAI can influence change, but also how to implement it effectively without harming trust or security.
As with other disruptive technologies, the current debate over GenAI is saturated with claims about impact. Again, technology alone won’t save healthcare. The real work is tackling deeper systemic issues, from misaligned incentives to fragmented workflows and the ever-present risk of undesirable outcomes.
GenAI shows great promise, but it also carries real risks. It’s not about whether we should use it, but how we can use it responsibly and with positive results.
The rise of shadow AI: innovation goes beyond surveillance
As GenAI tools proliferate, a new risk emerges: “shadow AI” – the use of generative AI by clinicians outside of institutional oversight. This is not a theoretical concern. When innovation outpaces governance, it opens the door to misinformation, “hallucinations,” and decisions that may seem right but are fundamentally wrong. For example, a question about the treatment of a complicated acute urinary tract infection treated on an outpatient basis would return fluoroquinolones. And this is a good answer IF the patient is not pregnant, because fluoroquinolones should be avoided during pregnancy due to risks to the fetus.
GenAI in clinical decision making must do more than just provide answers. It must clarify and define what the provider is asking and who is asking it to ensure the appropriate level of clinical nuance is represented.
The history of the healthcare industry is replete with examples of technology adoption going beyond readiness. We’ve seen it with EHRs, population health tools, and now with GenAI. The lesson is clear: experimentation does not equal preparation. Responsible leaders must not simply ask “can we deploy this?” » but “should we do it, and, if so, under what barriers?”
The Governance Gap: Why Oversight Matters
There is a governance gap between the value that GenAI can deliver and how it aligns and adheres to policies and guidelines that are intrinsic or regulated to it. If this gap is not monitored, the impact can be detrimental to care, potentially increasing disparities and eroding trust essential to health care.
Implementing governance is not a zero-sum action. This will not slow innovation or hinder deployment. Having clear guidelines can enable the creation of new solutions through clear policies that specify how applications and tools should be used, what data they should leverage, and what accountability looks like. Effective governance also means ensuring that infrastructure is secure, monitored, and enables detection of unsafe or unauthorized uses. AI tools should be integrated into secure and monitored environments, with real-time detection of unauthorized or unsafe use, including:
- Ensure clinicians are empowered through training not only on the value of its use, but also on its limitations and risks.
- A system understood of monitoring, feedback, and adaptation as a technology and clinical practice that is both ongoing and changing to include new tools and applications.
Separating Hype and Value
The path forward is clear: move from hype to value. The real test of GenAI lies not in its novelty, but in its ability to deliver measurable improvements for clinicians and patients. This means demanding a tangible return on investment: reduced administrative burden, better outcomes and an improved patient experience. It also means being honest about what GenAI cannot do and where human judgment remains irreplaceable.
Health sector leaders must insist on continuous and rigorous evaluation, transparent and regular reporting, and a focus on improvement. The goal is not just to adopt GenAI, but to integrate it in ways that support clinicians, empower patients, and advance health equity.
Leading the GenAI revolution
The GenAI revolution is here, but its success will depend not on the technology we select and use but on how we implement it. Will we allow shadow AI to thrive, will we close the governance gap and ensure innovation is our most important role?
The answer lies in strong leadership. Healthcare leaders must seek evidence-based tools, identify and work on clear policy, and be driven by a commitment to patient safety and trust. If we succeed, GenAI will not only transform healthcare workflows; this will help us build a more efficient, equitable and resilient health system for everyone.
Photo: Francescoch, Getty Images
Holly Urban, MD, MBA has extensive experience in healthcare technology and believes in the power of evidence-based content to transform EHRs beyond transactional systems into tools for clinicians to improve patient outcomes. After practicing as a primary care pediatrician, Dr. Urban worked for several EHR technology and evidence-based content companies, and held leadership roles in health IT for over fifteen years. Before joining Wolters Kluwer Healthshe served as CMIO at Oracle Cerner, Director of Product Management at MCG Health, and Vice President of Product Management at McKesson Horizon Clinicals.
This message appears via the MedCity Influencers program. Anyone can post their views on healthcare business and innovation on MedCity News through MedCity Influencers. Click here to find out how.