AI literacy for EMS providers, an evaluation framework built for emergencies, and the research, ethics and data practices behind our prototypes. Start here if you want to evaluate, collaborate or build.
A short module for students and providers. Use it alone or alongside the teaching resources.
Clinical judgment stays with the provider. AI output is an input to your assessment, never an order. You remain accountable for what you do, under your protocols and medical direction. Automation bias, the tendency to over-trust a machine suggestion, is a patient-safety risk, so verify before you act.
Your local protocol and medical direction. Treat the mismatch as an AI error and report it.
Treat it as unverified. Find the claim in an approved source before using it.
No. Absence of documentation is not evidence of absence. It is an information gap to confirm.
Where we are looking next, and where collaborators could fit. Everything below is planned work, not completed work.
| Study | Project | Question | Status |
|---|---|---|---|
| Extraction accuracy | Dispatch-to-Patient Needs | How well do extracted symptoms match expert annotations of de-identified narratives? | Planned |
| Gap-detection reliability | Patient Care Report Analyzer | What are the false-positive rates when flagging documentation gaps? | Planned |
| ETA baselines | Arrival Time to Hospital ML | Does the model beat historical-average and mapping-service baselines on time-based splits? | Planned |
| Measurement validation | CPR Analyzer | Do video-derived rate and interruption estimates agree with reference manikin sensors? | Planned |
| Larger protocol test set | Paramedic AI Copilot | Do results hold on a larger, harder set, and what caused the remaining errors? | Planned |
Where you could fit: clinician graders, de-identified data partners, statisticians, protocol owners, and EMS agencies for user testing. Tell us your interest.
General benchmarks miss emergencies. This framework builds on the evaluation design in our Paramedic AI Copilot report. Grading procedures marked "proposed" are the standard we intend for released sets.
| Task | Input | Expected output | Primary measures |
|---|---|---|---|
| Protocol Q&A | Question, optionally with retrieved protocol text | Answer with its source | Clinical correctness, protocol and source consistency |
| Medication reference | Drug question | Indication, contraindications, dosing reference | Medication accuracy, dose errors |
| ECG interpretation | ECG description or tracing | Rhythm and key findings | Interpretation accuracy |
| Scenario decision support | Simulated patient scenario | Priorities and recognition of life threats | Recognition, completeness, uncertainty handling |
| Documentation extraction | De-identified report text | Structured fields | Field accuracy, false-positive rate |
Each response is scored on: clinical correctness, completeness, protocol consistency, medication accuracy, ECG interpretation, recognition of life-threatening conditions, appropriate handling of uncertainty, and reference/source consistency. Each response is also checked for unsafe recommendations. Proposed scale: correct, partially correct, incorrect.
The unsafe-response rate is responses containing a predefined unsafe recommendation divided by all evaluated responses. Retrieval is scored separately (for example Recall@K).
Status: our training datasets are listed on the Datasets & Demos page. A documented evaluation benchmark has not been released yet. We plan a small, well-documented set of EMS scenarios (protocol questions, ECG interpretation, triage cases). The table below is our own internal prototype evaluation, not an open leaderboard.
| System | Test set | Overall accuracy | Notes |
|---|---|---|---|
| Paramedic AI Copilot, RAG | 100 simulated EMS scenarios | 92/100 (92%) | Medication 100%, ECG 91%, 0 unsafe responses |
| Same model, no retrieval | 100 simulated EMS scenarios | 71/100 (71%) | 29 incorrect responses vs 8 with RAG |
Simulated scenarios only. These results do not establish clinical efficacy or safety. Want to be listed once a public set exists? Email us.
Domain expertise from EMS clinicians and researchers.
Run your model on prehospital tasks using our rubric and error categories.
Clinician-written scenarios with reference answers and documentation.
Credentialed EMS reviewers grade outputs and explain failures.
Probe for unsafe doses, fabricated protocols and overconfidence.
Usability and trust studies with working providers [subject to ethics review].
One-page summaries. These describe opportunities, not deployed products.
A summary of our preliminary evaluation. Read the full report.
| Measure | With retrieval | Without retrieval |
|---|---|---|
| Correct responses | 92/100 | 71/100 |
| Incorrect responses | 8 | 29 |
| Medication scenarios | 100% accurate | 80% accurate |
| ECG scenarios | 91% accurate | 80% accurate |
| Unsafe responses identified | 0 | 0 |
More case studies will be added as evaluations are completed.
No versioned model releases yet. Each future release will list the model and version, knowledge-base version, and any evaluation rerun.
| When | Change |
|---|---|
| Oct 25, 2026 | Added AI Education Resources, Current Projects and this AI research page |
| Oct 25, 2026 | Added the Datasets & Demos page with dataset links, demo video and terms of use |
| Reported | Preliminary Paramedic AI Copilot evaluation on 100 simulated scenarios |
| Project | Next phases (planned) |
|---|---|
| CPR Analyzer | Validate against reference equipment, then real-time and sensor work |
| Patient Care Report Analyzer | Structured extraction and validation, completeness checks, protocol-grounded analysis |
| Arrival Time ML | Baseline comparisons on time-based splits, then geographic and traffic data |
| Dispatch Needs Analyzer | Structured classification, protocol-grounded analysis, pre-arrival briefing |
| Benchmarks | Publish a documented EMS scenario set with a comparison table |
See Current projects for each written plan.
