Download talk - Fraunhofer MEVIS

Software Support for
Oncological Therapy Response
Assessment
Frank Heckel, PhD
2015-07-13, Heidelberg Collaboratory for Image Processing
© Fraunhofer MEVIS
Bremen
Lübeck
Additional employees in Berlin, Leipzig, Heidelberg & Nijmegen
FRAUNHOFER MEVIS
1 / 52
© Fraunhofer MEVIS
Fraunhofer-Gesellschaft
 Largest organization for applied research in Europe
 Areas of research: life science, communication,
mobility, security, energy, environment
 66 institutes, 24.000 employees
 2.0 billion EUR research budget,
>70% from industry and public agencies
Basic Funding
Industry
Public Research
2 / 52
© Fraunhofer MEVIS
Fraunhofer-Gesellschaft
Itzehoe
Rostock
67 institutes in Germany
Bremen
Institutes
Branches, Working Groups,
Application Centers
Lübeck
Bremerhaven
Berlin
Potsdam
Hannover
Teltow
Nuthetal
Braunschweig
Magdeburg
Paderborn
Cottbus
Oberhausen
Halle
Dortmund
Leipzig
Schmallenberg
Schkopau
Duisburg
Dresden
Sankt Augustin
Aachen
Ilmenau Jena
Chemnitz
Euskirchen
Darmstadt
Würzburg
Kaiserslautern
St. Ingbert
Saarbrücken
Karlsruhe
Wertheim
Pfinztal
Erlangen
Nürnberg
Stuttgart
Freising
Freiburg
München
Efringen-Kirchen
3 / 52
© Fraunhofer MEVIS
Holzkirchen
Fraunhofer MEVIS
Non-profit
Commercial
(~100 employees)
Institute for
Medical Image Computing
Bremen
(~150 employees)
MeVis Medical Solutions AG
Bremen
(since 08/2007)
(since 01/2009)
51%
Project Group Image Registration
MeVis BreastCare GmbH & Co. KG
Lübeck
Bremen
4 / 52
© Fraunhofer MEVIS
(since 04/2010)
(since 10/2001)
Computer assistance for image-based,
personalized diagnosis and therapy
 Image acquisition and reconstruction
 Image computing, analysis and visualization
 Modelling and simulation
 Application, workflow and usability engineering
Solutions for clinical problems
5 / 52
© Fraunhofer MEVIS
Competences
Methods
MeVisLab
Validation
Organs
Navigation
Risk analysis
Bones/Joints
Visualization
Heart/Vessels
Quantification
Brain
Segmentation
Lung
Registration
Liver
Modeling/
Simulation
Imaging/
Modality
6 / 52
© Fraunhofer MEVIS
Clinical Workflow
Breast
Early Detection Diagnostic
Planning
Diagnosis
Therapy
Procedure
Monitoring
Organization Chart
Institute Directors
Prof. Kikinis, Prof. Hahn
Advisory Board
Employee Representatives
Extended Committee
Steering Committee
Administration
Steering Committee plus
representatives for:
Prof. Kikinis, Prof. Hahn,
T. Forstmann, Prof. Preußer,
Prof. Günther, Prof. Modersitzki,
Dr. Heldmann, Dr. Olesch,
Dr. Papenberg, Dr. Kraß,
Dr. Lang, Dr. Prause
T. Forstmann
Software/IT, QA, Employees,
Equal Rights, WTR, PR
7 / 52
© Fraunhofer MEVIS
Organization of Work
 Team-oriented
 Open-minded
 Self-organized
 Flexible
 Adaptive
8 / 52
© Fraunhofer MEVIS
Certification
Certificate for quality assurance
 Introduction and application of a quality
management system in compliance with
 EN ISO 9001 & EN ISO 13485 (medical devices)
 Since 2005 in Bremen
 Since 2012 in Lübeck
Scope:
 Research and development for computer assistance
of medical diagnosis and therapy
 Development and production of software for
medical products
9 / 52
© Fraunhofer MEVIS
Links to Universities
 University of Bremen
 Mathematics (H.-O. Peitgen, until Sep 2012)
 Medical Image Computing (R. Kikinis, since Jan 2014)
 MR Imaging & Physics (M. Günther)
 Jacobs University Bremen
 Analysis & Visualization (H. Hahn)
 Modeling & Simulation (T. Preußer)
 University of Lübeck
 Mathematics & MEVIS Project Group
(B. Fischer †, J. Modersitzki)
 University of Nijmegen
 Computer-Aided Detection & Diagnosis
(N. Karssemeijer, B. van Ginneken)
10 / 52
© Fraunhofer MEVIS
INNOVATION CENTER COMPUTER
ASSISTED SURGERY (ICCAS)
11 / 52
© Fraunhofer MEVIS
Innovation Center Computer Assisted Surgery (ICCAS)
Part of medical faculty Universität Leipzig
Clinical disciplines: ENT-surgery, Heart surgery, Neurosurgery
12 / 52
© Fraunhofer MEVIS
ICCAS Research Areas
MAI
DPM
STD
MAI – Model-based automation and integration, DPM – Digital patient model, STD - Standardization
13 / 52
© Fraunhofer MEVIS
Research Area: Model-based Automation and
Integration
Head: Prof. Thomas Neumuth
Augmented
Reality for
microscopes
Navigation
data
Model
visualisations
System
monitoring
Ultrasound
imaging
Tracked
ultrasound
probe
Information and communication technology in the OR
14 / 52
© Fraunhofer MEVIS
Research Area: Model-based Automation and
Integration
Surgical
Workflow
patient
surgeon
HMI
15 / 52
© Fraunhofer MEVIS
Imaging
Navigation
Research Area: Model-based Automation and
Integration
Integration into therapeutic process
Workflow management
Data consolidation and integration
Process monitoring
16 / 52
© Fraunhofer MEVIS
Ressource monitoring
Research Area: Digital Patient Models
Head: Dr. Kerstin Denecke
17 / 52
© Fraunhofer MEVIS
Research Area: Standardization
Head: Prof. Heinz Lemke
18 / 52
© Fraunhofer MEVIS
Research Area: Image-guided Interventions
Head (and Insitute Director): Prof. Andreas Melzer
19 / 52
© Fraunhofer MEVIS
ONCOLOGICAL THERAPY
RESPONSE ASSESSMENT
20 / 52
© Fraunhofer MEVIS
Overview
 Background
 Semi-Automatic Segmentation
 Segmentation Editing
 Partial Volume Correction
 The Ground Truth Problem
 Workflow Aspects
21 / 52
© Fraunhofer MEVIS
Background
Cancer and Chemotherapy
 Cause for 13% of all deaths worldwide
 Every 2nd man gets cancer  every 4th dies
 Treatment examples:
 Surgery
 Radiotherapy
 Radiofrequency ablation and …
 Chemotherapy
 Lung nodules, metastases, enlarged lymph nodes
 Systemic treatment
 Severe side effects
 Different agents
22 / 52
© Fraunhofer MEVIS
Background
3-6 months
CT-Based Follow-Up Examination
Baseline
1st Follow-Up
•Find tumors
•Identify target lesions
•Measure target lesions
•Reporting
•Find target lesions
•Measure response
•Look for new lesions
•Reporting
23 / 52
© Fraunhofer MEVIS
Additional FollowUps
Background
Oncological Therapy Response Monitoring
 Change in tumor size is an important criterion
 RECIST1 1.1: Sum of maximum diameters of target lesions  Relative
change
Complete
Response
Partial
Response
Stable
Disease
Progressive
Disease
Disappearance
< -30%
-30% – 20%
> +20%
 Volume is a more accurate measure
 Many tumors grow/shrink irregularly in 3D
 Requires appropriate segmentation
 Progress/response not defined
 Not used in clinical routine
1
RECIST: Response Evaluation Criteria In Solid Tumors
24 / 52
© Fraunhofer MEVIS
Background
Diameter vs. Volume
Classification
Diameter
Volume
25 / 52
© Fraunhofer MEVIS
stable
disease
progressive
disease
partial
response
complete
response
Small change
> +20%
< -30%
no longer
visible
> + 73%
< -66%
Background
Robustness of Diameter Measurement
 Simulated example:
 Measured 2% change
 Reality: 26% change (roughly double volume!)
26 / 52
© Fraunhofer MEVIS
The Segmentation Problem
 Ultimate Goal:
Automatic segmentation for a wide range of objects
 Reproducible results with no effort for the user
 Solutions for specific purposes
 Might fail (low contrast, noise, biological variability)
 Unsolved or insufficient for many real-world problems
 Alternatives:
 Manual segmentation
 Semi-automatic or interactive tools
 (Semi-)automatic algorithm followed by manual correction
 Drawback: Variability due to different inputs or judgment
27 / 52
© Fraunhofer MEVIS
Semi-Automatic Segmentation
 Familiar user Interaction:
draw the (maximum) diameter
 Core method: “Smart Opening”1
 Region Growing
 Erosion
 Dilation
 Refinement
 Specific variation for lung nodules, liver metastases and lymph nodes2
 For lymph nodes a spiral-scanning solution
has been developed as well3
1
2
3
Kuhnigk et al., IEEE TMI, 25(4), 2006
Moltz et al., IEEE Journal of Selected Topics in Signal Processing, 3(1), 2009
Wang et al., SPIE Medical Imaging, 2012
28 / 52
© Fraunhofer MEVIS
Semi-Automatic Segmentation
Examples for Challenging Lung Nodules
29 / 52
© Fraunhofer MEVIS
Semi-Automatic Segmentation
Examples for Challenging Liver Metastases
 Positive examples:
 Negative examples:
30 / 52
© Fraunhofer MEVIS
Semi-Automatic Segmentation
Examples for Challenging Lymph Nodes
 Smart Opening (top) vs. Spiral Scanning (bottom)
31 / 52
© Fraunhofer MEVIS
Semi-Automatic Segmentation
Evaluation
 Lung: LIDC-Data (674 cases (solid nodules), 4 reference segmentations)
 Liver: MDS-Data (371 cases, 1 reference segmentation)
Volume overlap
Hausdorff distance
Computation time
Lung
68,3%
2,46 mm
0,41 s
Liver
62,6%
4,20 mm
0,75 s
32 / 52
© Fraunhofer MEVIS
Semi-Automatic Segmentation
Evaluation
 Clinical Evaluation: Amount of Lesions that have not been manually corrected
Lung
Liver
100
100
90
90
80
80
70
70
60
60
50
50
40
40
30
30
20
20
10
10
0
0
2009
33 / 52
© Fraunhofer MEVIS
2010
2012
2011
2012
Semi-Automatic Segmentation
Evaluation
 Clinical Evaluation: Amount of Lesions that have not been manually corrected
Lymph nodes
100
90
80
70
60
50
40
30
20
10
0
2008
34 / 52
© Fraunhofer MEVIS
2010
2010
2010
2011
2012
2012
Segmentation Editing
Stop
Segmentation Algorithm
Automatic
yes
Segmentation
Result
yes
Satisfying?
no
Initial
Algorithm allows
modification?
Semi-automatic
Start
no
Segmentation Algorithm
Interactive
Segmentation
Result
Satisfying?
Stop
yes
 Most existing methods are low-level and unintuitive in 3D
 High-level correction has not received much attention in research
35 / 52
© Fraunhofer MEVIS
no
Segmentation
Editing Algorithm
Segmentation Editing
Sketch-Based Editing in 2D
add
remove
add +
remove
replace
36 / 52
© Fraunhofer MEVIS
Segmentation Editing
3D Extrapolation
 Image-based method (→ shortest path)
 Image-independent method (→ RBF-based 3D object reconstruction)
Heckel et al., Computer Graphics Forum, 32(8), 2013
37 / 52
© Fraunhofer MEVIS
Segmentation Editing
Qualitative Evaluation
 131 representative tumor segmentations in CT (lung nodules, liver
metastases, lymph nodes)
 5 radiologists with different level of experience
 Editing rating score:
𝑟edit =
Heckel et al., SPIE Journal of Medical Imaging, 1(3), 2014
38 / 52
© Fraunhofer MEVIS
1
0.0𝑟−− + 0.25𝑟− + 0.5𝑟0 + 0.75𝑟+ + 1.0𝑟++
𝑁
Segmentation Editing
Quantitative Evaluation
 Analyze quality over time
 Editing quality score: 𝑚edit,𝑠max =
39 / 52
© Fraunhofer MEVIS
1
𝑆max
min(𝑆,𝑆max )
𝑚𝑖
𝑖=1
+ 𝑆 ∙ 𝑚𝑆
Segmentation Editing
Simulation-Based Evaluation
 Problem: High effort and bad reproducibility of user studies
 Idea: Replace user by a simulation
 Benefits:
 Objective and reproducible validation
Stop
yes
Start
Validation
Satisfying?
no
 Objective comparison
 Improved regression testing
Intermediate
Segmentation
Reference
Target
Segmentation
 Better parameter tuning
Simulation
User
User
Input
Segmentation
Editing
40 / 52
© Fraunhofer MEVIS
Previous
Inputs
Control flow
Data flow
Segmentation Editing
Simulation-Based Evaluation
 Step 1: Find most probably corrected 3D error
 Step 2: Select slice and view where the error is most probably corrected
 Step 3: Generate user-input for sketching
 Step 4: Apply editing algorithm
Heckel et al., Scandinavian Conferences on Image Analysis, 2013
41 / 52
© Fraunhofer MEVIS
Segmentation Editing
Simulation-Based Evaluation
42 / 52
© Fraunhofer MEVIS
Partial Volume Correction
The Partial Volume Effect
 Smoothing effect caused by limited spatial resolution (of CT)
 Ill-defined border between tumor and healthy tissue, making segmentation an
ill-defined problem
 Could cause significant differences in size measurements
28.4 ml
(-27.5%)
43 / 52
© Fraunhofer MEVIS
39.2 ml
56.8 ml
(+44.9%)
Partial Volume Correction
Method
 Spatial subdivision into spherical sectors
to cover different tissues
 Define reference tissue values inside and
outside of the object (𝑡𝑖 and to) per sector
 For each sector 𝑠: compute the weight w
of each partial volume voxel
𝑡𝑜 − 𝑣
𝑤 𝑉 = 𝑠
, 𝑉 ∈ 𝑃𝑖𝑠 ∪ 𝑃𝑜𝑠
𝑡𝑜𝑠 − 𝑡𝑖𝑠
1.0
0.75
0.5
𝑉𝑜𝑙𝐿 =
𝑤 𝑉 𝑉𝑜𝑙𝑉
𝑉∈𝐿
Heckel et al., IEEE TMI, 33(2), 2014
44 / 52
© Fraunhofer MEVIS
0.25
70.8 ml
71.1 ml
0.0
Partial Volume Correction
Software Phantom Results
45 / 52
© Fraunhofer MEVIS
Partial Volume Correction
Hardware Phantom Results
46 / 52
© Fraunhofer MEVIS
Partial Volume Correction
Multi-Reader Data Results
47 / 52
© Fraunhofer MEVIS
The Ground Truth Problem
There is no „Ground Truth“!
 Expert segmentations differ significantly
 Variability depends on several aspects
(lesion size, contrast, partial volume effects, interpretation, …)
 We need to consider n>1 reference segmentations
 Who are experts? Only clinicians?
Jan Moltz, PhD Thesis, Jacobs University Bremen, 2013
48 / 52
© Fraunhofer MEVIS
The Ground Truth Problem
What is a „good“ segmentation result?
49 / 52
© Fraunhofer MEVIS
Workflow Aspects
 CAD
 Lesion Matching
 Visualization
 Reporting
Schwier et al., IJCARS, 6(6), 2011
Schwier et al., CARS 2009
50 / 52
© Fraunhofer MEVIS
Jan Moltz et al., ISBI, 2009
Workflow Aspects
Prototyping
51 / 52
© Fraunhofer MEVIS
Thank you!
frank.heckel@mevis.fraunhofer.de
52 / 52
© Fraunhofer MEVIS
53 / 52
© Fraunhofer MEVIS