Download Title goes here

Curriculum Inventory Administrators’ Group
August 10, 2016
Terri Cameron, MA
Director, Curriculum Programs
Agenda
• 2015-2016 Upload | Key Documentation Issues
• August CI in Context: Grading Systems in US Medical Schools (‘Letter’ grades vs P/F)
• September CI in Context: Curricula Regarding Care of Veterans
• Featured Report: Volunteer Faculty Retention
• Updated IPEC Competencies – Effect on PCRS
• January – August Report Requests
• Annual Evaluation / Frequency of CIAG sessions
• Curriculum Inventory Task Force / CI Research Group
• CI and MedEdPORTAL at Learn Serve Lead
• Planning for Spring 2017 Education Meetings
• Natural Language Processing at Vanderbilt University
• Medical School Highlight: University of North Carolina
• Next meeting: Wednesday, September 14, 1 pm ET
2015-2016 Upload
• 14 Schools Submitted; 3 in Review; 1 Rejected; 4 Verified
• Lessons from 2014-2015 Data:
• Verification Report Review – please check Academic Level dates –
they are calculated from Sequence Block Start and End Dates
• Clerkship Dates – Please make sure Clerkship Start and End Dates
reflect the entire period that the Clerkship is offered (e.g., July 1,
2015 – June 30, 2016), rather than for a single iteration
• Sequence Blocks with same start and end dates?
• 0 hours duration for Events?
• HTML Codes in text (copying from web sites)
• Exceeding 4,000 character limit?
Key Documentation Issues for 2015-2016 Upload
•
Using Program Description to Designate Approval of School Name
linked to CI Reports
• Use Program Description to include the text:
• ReleaseName=Y OR
• ReleaseName=N
• For systems that do not have this option, an e-mail can be
sent to ci@aamc.org (e-mail will need to be sent each year)
• Documenting Gateway Exams
• Documenting USMLE Preparation
• Use updated Resource List using IDs if possible; text if not
CI in Context:
August 2016
• Grading Systems in US
Medical Schools
• Cynthia Ledford, MD
The Ohio State
University
CI in Context:
September 2016
30
25
25
Number of Occurrences
• Curricula Regarding Care
of Veterans
• Monica Lypson, MD
University of Michigan
Summary of Reported Occurrences* of Veteran Care in Medical School
Curricula
20
15
10
10
5
3
2
0
Max Course
Median Course
Max Teaching Events
Occurrences: Courses/Clerkships and Teaching Events
Median Teaching Events
Featured CI Report:
Volunteer Faculty Retention
Updated IPEC Competencies
•
Updated IPEC Competencies Released in June (updated link):
https://ipecollaborative.org/uploads/IPEC-2016-Updated-CoreCompetencies-Report__final_release_.PDF
•
Effect on PCRS
Reports Requested Jan – August 2016
Adv Childhood Exp
Anatomy
Aspirin
Behavioral Science
Biochemistry
Biostatistics
Breast Exam
CEPAER
Cell Biology
Climate Change
Clinical Diagnosis
Eating Disorders
Elder Abuse
Embryology
Endocrine
Environmental Medicine
Epidemiology
Ethics
EBM
Genetics
Geriatrics
Giving Bad News
History of Medicine
Immunology
IPE
LGBT
Medical Marijuana
Medical Socioeconomics
Mental Health
Microbiology
Mindfullness
Musculoskeletal
Neuroscience
Pathology
Pathophysiology
Patient Safety
Pelvic Exam
Pharmacology
Physiatry
Physical Medicine
Physician Impairment
Physiology
Population Health
Public Health Issues
Clinical Skills
Comm / Ethics
Community Health
Comm Service
Correctional Health
Disability
Disclosure
Early Clin Exp
Gun Safety
Health Care Financing
Health Care Quality Improv
Health Care Reform
Health Care Systems
Health Disparities
Health Policy
Histology
Nutrition
Obesity
Ophthalmology
Opioids
Organ System
Orthopaedics
Pain Management
Palliative Care
Radiology
Rheumatology
Service Learning
Simulation
Sports Medicine
Stand/Virtual Pt
Substance Abuse
Veteran Care
Annual Evaluation / Frequency of CIAG sessions
•
Will be sending out Annual Evaluation of CIAG sessions
• Other metrics for evaluating?
• Participation
• Slides/recordings downloaded
•
Major issue: Frequency of CIAG Sessions
• Monthly?
• Bi-monthly?
• Quarterly?
CI Task Force
Update:
•
Seeking two additional members
•
Next Meeting is August 24
CI Research Group
• Has developed a list of potential research projects
• Will be reviewing data needs
• Retreat in September
CI and MedEdPORTAL at Learn Serve Lead
•
Curriculum Inventory Update
• November 13, 2016, 4:30-5:45PM, Center 618-620
•
Between the Bench and the Bedside: The Role of Scholarly
Publishing in Advancing Academic Medicine
• November 14, 2016, 10:30-11:45AM, Center: 611-614
•
MedEdPORTAL/Academic Medicine Reviewer Reception and
Award Ceremony
• November 14, 2016, 7:30-9PM, Hyatt Leonesa I and II
CI at 2017 Medical Education Meetings
•
Seeking schools with to present at Spring/Summer 2017 meetings (complete list
will be posted to website in near future):
•
Innovative approach to curriculum documentation
• Faculty Development
• Technical Process
• Data Collection Process
•
Use of Curriculum Inventory Resources
• Support of Curriculum Committee(s)
•
Benchmarking
•
Continuous Quality Improvement
• Support of Faculty
•
Research
•
Collaboration
2015-2016 CI Upload
•
Portal opens August 1; closes September 30.
•
Staging open now
•
Goals:
• US medical schools: 100 percent participation (148)
• Canadian medical schools: 50 percent participation (9)
• Osteopathic School Pilot: 30 percent participation (10)
Using Natural Language Processing to Make
Content Recommendations
and
Identify Knowledge Gaps in Curriculum
Toufeeq Ahmed, PhD, MS (Vanderbilt)
Assistant Dean of Education Informatics
Assistant Professor of BioMedical Informatics
Pedro Teixeira, PhD (Vanderbilt)
Tao Le, MD, MPH (USMLE-Rx)
Natural Language Processing (NLP) extracts
higher-level concepts
• Free text has high expressivity
• A single concept can be expressed with many
different words and characters
• Structured sets of concepts are catalogued in
medical terminologies e.g. (Unified Medical
Language System - UMLS)
• NLP maps many variants to a single concept:
“myocardial infarction”, “MI”, “heart attack”  one concept
Many ways to write “Diabetes mellitus type 2”
18
NLP quickly extracts concepts from
document text
hypertension
Text A
multiple sclerosis
Text B
Text C
NLP
Engine
joint stiffness
muscle spasticity
type 2 diabetes
Input patient
note
Concept mapped
patient note
20
Concept Mapping Result and Application
to Resource Recommendation
Concept Mapping Result and Application
to Resource Recommendation
QuickMatch
• Light-weight dictionary-based
• Implemented in Python
• Easily-scalable webservice (returns results as JSON)
• Includes normalization by capitalization, white space, and
grammatical variants
• Ranks results by “value”
• Calculated using inverse of term frequency, match quality, and overall concept
specificity
• Dictionary is based on medical terms filtered from Wikipedia
QuickMatch’s Wikipedia-based Dictionary
• 70,799 total unique medical concepts
• Filtered using medical terminology references and medical keywords
• Supplemented with cross-match to Systematized Nomenclature of
Medicine-Clinical Terms (SNOMED-CT)
Wikipedia Miner
• Open-source toolkit to easily leverage Wikipedia
• Includes: concept annotation, search, suggestion, and article
exploration/comparison
• Text preprocessed for HTML and MediaWiki markup
• Annotation accomplished using:
• topic detection, disambiguation (machine learning-based), link detection (for
ranking topic significance), and applies the resulting “tags” to documents
Milne D, Witten IH. An open-source toolkit for mining Wikipedia. AI. 2013 Jan;
Available: http://www.sciencedirect.com/science/article/pii/S000437021200077X
The Mayo clinical Text Analysis and
Knowledge Extraction System (cTAKES)
• Modular open source NLP system for unstructured clinical text
• Implemented in Java
• Modules Include:
• sentence boundary detection, tokenization, normalization, part of speech
tagging, shallow parsing, named entity recognition, annotation, and
negation/status detection
• Uses Unified Medical Language System (UMLS)-based dictionary lookup
Savova et al. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES):
architecture, component evaluation and applications. J Am Med Inform Assoc. 2010
Evaluation Dataset
• 50 question learning point summaries
• Manually annotated for medical concepts
• Concepts classified as low or high yield
• High yield concepts = items a learner would likely look
up
• Includes:
• diseases, medications, and procedures (Takotsubo cardiomyopathy,
Lisinopril, paracentesis)
• Excludes:
• common anatomy, symptoms, etc. (heart, pain, blood vessel, etc.)
cTAKES has the highest recall for
any match (All Concepts)
Method
QuickMatch
Wikipedia Miner
cTAKES
Total
Any Recall Recall
Match (Exact) (Any)
85
236 0.535 0.747
19
240 0.579 0.759
55
252 0.513 0.797
Exact Partial Extra
316
316
316
169
183
162
137
111
90
Wikipedia Miner has the highest recall (Highyield Concepts)
Method
QuickMatch
Wikipedia Miner
cTAKES
Total
Any Recall Recall
Match (Exact) (Any)
0
141 0.626 0.788
0
159 0.626 0.888
0
142 0.497 0.793
Exact Partial Extra
179
179
179
112
112
89
89
85
53
QuickMatch is Faster than Wikipedia
Miner (30x) and cTAKES (140x)
Concept Mapping For Content
Recommendation
Myocardial
Infarction
Curriculum Coverage (Benchmarking)
NLP for Semantic Curriculum Search
NLP for Semantic Curriculum Search
Thank you!
Toufeeq Ahmed, PhD MS
toufeeq.ahmed@Vanderbilt.Edu
TEC Curriculum
Translational Education at Carolina
Kurt Gilliland, PhD
Assistant Dean for Curriculum and Evaluation
University of North Carolina School of Medicine
Chapel Hill, North Carolina
Why a new curriculum?
Health care and medical education
are always changing.
Expanded quantity of
information in science and
medicine
Need for more opportunity
for individualization
Difference in
Millennial
learners
Changing nature of health
care
Pressures on timeline for
career choice
(earlier residency
application)
Translating what doctors need to know
TEC1
TEC2
“Translation of Medical Science
to People”
“Translation of Medical Science
to Patient Centered Care”
Carol Otey
Cell Biology and Physiology
Cristin Colford
Medicine
TEC1
TEC1
TEC2
TEC2
TEC3
TEC3
PD
PD
TEC3
Professional Development (PD)
“Translation of Medical Science
to the Care of Populations”
Anthony Viera
Family Medicine
Sue Estroff
Social Medicine
TEC Curriculum Overview
Aug
2016
Foundation
Phase
Application
Phase
Individualization
Phase
(17 months)
(12 months)
(14 months)
Dec Mar
2017 2018
M.O. = Multi-organ Synthesis Block
Mar
2019
May
2020
A Sample Block: Urinary System
Nephrology
and
Urology
Feb 1
Urinary System Block
Mar 1
Urinary System Block
Nephrology
and
Urology
Anatomy, Histology,
& Embryology
Feb 1
Physiology,
Biochemistry &
Genetics
Radiology
Epidemiology
Urinary System Block
Nutrition
Pathology &
Microbiology
Mar 1
Pharmacology
Matrix of Leadership
• Course and block directors look at time on the calendar
• Thread and Coil directors work longitudinally
• Lots of faculty involved, lots of time spent integrating
Block Directors
Coils
Students see relevance
UNC Midpoint Survey
Students feel that they are learning what they need to
learn in order to become competent physicians.
Students can see the relationship between what they
are studying and the kinds of patient care situations
they will meet when they graduate.
Courses emphasize the interdependence of facts,
concepts and principles.
The relationship between basic science and clinical
material is made clear. *
2016 TEC
2015 PreTec
Exams emphasize understanding of concepts rather
than memorization of facts.
Courses do not emphasize memorization of trivial
details. *
1
2
3
4
5
(1, never; 2, seldom; 3, occasionally; 4, fairly often; 5, very often)
Student Satisfaction
(1, never; 2, seldom; 3, occasionally; 4, fairly often; 5, very often)
Curriculum Mapping at UNC
UNC Core Competencies
Medical knowledge  Clinical skills 
Communication  Professionalism 
Lifelong learning  Systems-based practice
Population health
Program
Objectives
Course
Objectives
Block Objectives
Session-level Objectives
These are derived from the UNC
core competencies.
These are items from the USMLE
content guideline.
These are written by the instructor.
Curriculum Mapping at UNC
2014-15 (a year ago)
Session-level objectives
were “harvested” from
PPTs and other session
documents.
Mapping of sessions to…
• Block objectives
(USMLE content) and
LCME hot topics
• Course objectives
and program
objectives
one45
This was done by Dale Krams
and Kurt Gilliland.
One45: a software packaged developed in British Columbia for managing,
scheduling, assessing, and tracking
Thank you to Dale Krams of the UNC Office of Medical Education for coordinating the process.
Curriculum Mapping at UNC
2015-16 (this past year)
Mapping of sessions to…
• Block objectives
(USMLE content) and
LCME hot topics
• Course objectives
and program
objectives
This was done by faculty
course leaders using paper
in a notebook.
one45
Gap analysis
One45: a software packaged developed in British Columbia for managing,
scheduling, assessing, and tracking
Thank you to Dale Krams of the UNC Office of Medical Education for coordinating the process.
TEC Curriculum Overview
Foundation
Phase
Application
Phase
Individualization
Phase
(17 months)
(12 months)
(14 months)
Aug
2016
Dec Mar
2017 2018
Mar
2019
Community Based
Longitudinal Care
Care of Specific
Populations
(16 weeks)
(16 weeks)
• Adult and pediatric
ambulatory clinics
• Neurology
• Psychiatry
• Obstetrics
• Pediatrics (inpatient)
May
2020
Hospital,
Interventional, and
Surgical Care
(16 weeks)
• Surgery and gynecology
• Medicine (inpatient)
Intensive Integration: Monthly, day-long sessions, where all students convene
in Chapel Hill for small-group sessions and large group presentations
Asheville
NC Area Health Education Centers (AHEC)
Greensboro
Raleigh
Chapel Hill
Charlotte
Charlotte
Wilmington
TEC Curriculum Overview
Aug
2016
Foundation
Phase
Application
Phase
Individualization
Phase
(17 months)
(12 months)
(14 months)
Dec Mar
2017 2018
APS = Advanced Practice Selective
Mar
2019
May
2020
Questions?
Next meeting:
•
Wednesday, September 14, 1 pm ET
• (Second Wednesday of each month, 1 pm ET)
• Registration Links posted in Training and Resources section of
www.aamc.org/cir
• Please send agenda items to tcameron@aamc.org