Background
Hospitals manage safety incidents every day, from medication errors to workplace violence. Incident owners are the people responsible for reviewing these reports. They investigate what happened and look for ways to prevent similar events.
My initial work with an Upstate New York health system focused on helping employees report incidents using AI. I also designed a North Star Copilot for the safety teams reviewing those reports, extending the product vision from better reporting to AI-assisted analysis and review.
The design centered on a question: How much should an agent do independently, and where does the user need visibility or control?
Impact
Secured client buy-in by setting the vision for AI-powered incident management
The North Star was a major reason the client committed to the initial intake work. I made AI’s potential tangible: how it could support investigations, what it could do independently, and where people would retain control. Demonstrating both the value and the interaction model built confidence in the broader product vision.
Related case study: Finding the Right Place for AI in Hospital Safety
Personalize the assistance, explain the choices
A safety team member starts with a question: “Where is workplace violence increasing?”
Copilot personalizes the analysis using the user’s previous behavior. Here, it selects a 90-day timeframe based on how they typically review safety trends and explains that choice in chat. The user can adjust it at any point. When more information is needed, a compact form helps establish the scope.
A starting point the user can change
Asking users to specify every parameter would slow down familiar work. Using a default without explaining it could hide an important assumption. I designed the timeframe to be visible and adjustable, so personalization reduces setup while keeping the scope clear.
The conversation shows progress and important decisions. A dedicated canvas presents the completed analysis, giving the user a clear place to examine the result.
Design principle: Use context to reduce setup, and explain consequential choices. Personalization should give users a useful starting point they can understand and change.
Build on previous analysis
One finding often leads to another question. Users can ask for a breakdown by work shift, adjust the timeframe, or explore a related comparison.
Each session keeps the conversation and its analyses together. Users can switch between results, pin useful outputs, and return to previous work without starting over.
Design principle: Preserve context across questions. Keep related analyses and their conversation connected so users can build on previous work.
Build trust through inspectable evidence
Building trust in an AI-generated finding starts with being able to inspect the evidence behind it.
A source preview connects each analysis to the underlying incident reports. Users can open the relevant records in the existing incident-management application, then return to the analysis without losing their place.
This keeps supporting evidence within reach and connects the analysis to the tools safety teams already use.
Design principle: Make evidence part of the interaction. Give users a direct path from a finding to its supporting records and back to the analysis.
Ask before changing the scope
A comparison across hospitals can become misleading if one hospital’s data is missing.
Copilot explains the retrieval problem and retries automatically. If the retry fails, it pauses and lets the user decide whether to try again or continue without that hospital.
If the user chooses to continue, the completed analysis identifies the excluded hospital. The change remains visible wherever the result is reviewed.
Design principle: Match autonomy to consequence. Let the agent retry on its own, but ask before leaving out a hospital and changing the comparison. Keep that exclusion visible in the result so people can understand it later.
Connect incident patterns with organizational guidance
Incident reports describe what happened. Shared policies and procedures provide context for what the organization expects. I designed Copilot to bring these sources together, helping safety teams identify areas for further review.
Copilot finds relevant guidance in shared organizational documents and explains which sources it uses. The comparison connects findings to incident reports and specific guidance, making the knowledge behind the response visible and inspectable.
Users can add, replace, or remove reference documents and rerun the comparison. The revised result reflects those changes while preserving the earlier response and its evidence.
Keeping earlier results understandable
Changing a reference document changes the basis for the next response. I designed the revised comparison to preserve the earlier response and its evidence, so users can inspect what each version relied on.
Design principle: Make shared knowledge visible and correctable. Let users inspect and change the sources informing a response, while preserving the context behind earlier results.
Evaluating agent behavior
My approach is to give the agent a clear task, introduce a problem or change, and check how it responds. Does it follow the user’s instructions, support its answers with evidence, and recover when something goes wrong?
| What I check | What good behavior looks like |
|---|---|
| Keeps the user in control | If one hospital’s data is missing, the agent asks before leaving it out. If the user says no, it stops. |
| Gives accurate answers | Compare the answer with the underlying records. The numbers should be correct, and the evidence should support the conclusion. |
| Recognizes missing evidence | When the records cannot answer the question, the agent explains what is missing instead of guessing. |
| Recovers without losing work | If an update fails, the earlier result stays available. The agent explains what failed and offers a next step. |
| Tracks changes to sources | When guidance changes, the next answer uses the new version. Earlier answers stay connected to the sources they used. |
| Makes exclusions clear | If the user agrees to leave out a hospital, that exclusion stays visible in the result, including when it is reopened. |
Moving the work forward, keeping people in control
This work connected an achievable first release to an ambitious product vision. I raised the bar for the experience by showing how AI could help safety teams explore patterns, review evidence, and decide where to investigate further.
My contribution extended from identifying where AI could help to defining how an agent should behave: when to act, what to explain, how to preserve evidence, and when to return a decision to the user.
The result combines personalized assistance with visible progress, inspectable evidence, and human-in-the-loop recovery. Safety teams can move their analysis forward while staying in control of the work.