Background: Picture archiving and communication systems (PACS) are ubiquitously used to store, share, and view radiological information for preoperative planning across surgical specialties. Although traditional PACS software has proven reliable in terms of display accuracy and ease of use, it remains limited by its inherent representation of medical imaging in 2 dimensions. Augmented reality (AR) systems present an exciting opportunity to complement traditional PACS capabilities.
Objective: This study aims to evaluate the technical feasibility of using a novel AR platform, with holograms derived from computed tomography (CT) imaging, as a supplement to traditional PACS for presurgical planning in complex surgical procedures.
Methods: Independent readers measured objects of predetermined, anthropomorphically correlated sizes using the circumference and angle tools of standard-of-care PACS software and a newly developed augmented reality presurgical planning system (ARPPS).
Results: Measurements taken with the standard PACS and the ARPPS showed no statistically significant differences. Bland-Altman analysis showed a mean difference of 0.08% (95% CI –4.20% to 4.36%) for measurements taken with PACS versus ARPPS’ circumference tools and –1.84% (95% CI –6.17% to 2.14%) for measurements with the systems’ angle tools. Lin’s concordance correlation coefficients were 1.00 and 0.98 for the circumference and angle measurements, respectively, indicating almost perfect strength of agreement between ARPPS and PACS. Intraclass correlation showed no statistically significant difference between the readers for either measurement tool on each system.
Conclusions: ARPPS can be an effective, accurate, and precise means of 3D visualization and measurement of CT-derived holograms in the presurgical care timeline.
Picture archiving and communication systems (PACS) allow for easy storage and viewing of medical imaging information. Traditional PACS viewers present images in x-ray, computed tomography (CT), and magnetic resonance imaging (MRI) data on a 2-dimensional (2D) workstation screen to be examined by a surgical team in preparation for a complex procedure [, ]. While these systems have been shown to be accurate and easy to use for the analysis of medical images [ ], they are also limited by their requirement of a desktop computer, laptop, or smartphone screen [ ]. Dias et al [ ] report that 2 of the most common problems of traditional PACS are the mismatch between the 2D viewing screen and the real world and the accompanying lack of flexibility and efficiency of use.
Augmented reality (AR) and virtual reality (VR) technologies have the potential to address these shortcomings. AR and VR alike allow for the realistic and interactive digital representation of objects in a 3D space. As such, both technologies are already successfully deployed across a diverse set of applications, including terrestrial navigation , architectural modeling [ ], automotive engineering [ ], and education [ ]. The same properties could be applied to present a realistic overlay of medical devices and tools on patients’ anatomy in 3D space on a portable, shared visualization method.
Whereas VR presents an entirely digital representation of objects and their environment, AR allows for the overlay of digital holograms on a live real-world scene. In addition, many VR systems require a dedicated physical play space to allow for the experience of the completely immersive digital experience . These characteristics make AR a more likely candidate for the development of interactive tools assisting the dynamic clinical workflow.
The potential of AR systems to assist in clinical tasks has been extensively reviewed by Uppot et al . Possible use cases include supplementing radiology training; communicating with colleagues, referring clinicians, and patients; and aiding in interventional radiology procedures. Additional uses for AR in medicine include providing simulations for advanced life support training [ ], visualizing patient anatomy including tumors [ ], and guiding assistants during robotic surgery [ ]. The increased spatial understanding of anatomy with AR has been shown to positively impact surgical care during laparoscopic surgery for visualizing hidden patient anatomy [ ], resection of neurological tumors without causing new neurological deficit [ ], and breast tumor resection by maximizing breast conservation [ ]. Multiple other non–patient outcome benefits have been proposed, including overall operating room efficiency [ , ], and more specifically—reduced operating room time, increased surgical precision, and reduced radiation exposure [ ].
In order to create an AR model suitable for presurgical planning, the medical image from a CT or MRI scan must first be segmented using a DICOM viewer to visualize only the object or organ of interest. The resulting image is passed onto an image processing software that renders the object’s volumes and surfaces into a 3D scalar field model. This model can later be loaded in a dedicated AR software designed for projecting the image onto an AR or mixed reality headset display. Similar technologies have evaluated the use of AR systems for the visualization of MRI data . However, the focus of this study is the validation of CT-derived holograms. Although the visualization of CT-derived holograms has been assessed, measurement systems for these CT-derived holograms are rarely evaluated or utilized.
As AR becomes more widely used in presurgical planning, it is crucial to know that these systems meet the gold standard for medical image measurement. This study aims to validate the feasibility, safety, and efficacy of a novel ARPPS, compared to a standard-of-care PACS viewer, in order to support its use in the presurgical visualization and measurement of CT-derived imaging of patient anatomy and surgical tools.
A CT image data set was generated using Discovery CT750 HD (GE Healthcare). The object imaged was a CT dose meter phantom (model 137856101, GE Healthcare) compliant with the American College of Radiology standards. The PACS used for standard-of-care comparison was Osirix MD version 10.0 (Pixmeo SARL; FDA 510(k) K101342) . The experimental PACS was the RadHA ARPPS version 3.3 (University of California, San Francisco) ( ), as viewed on HoloLens generation 1 headset (Microsoft Corp). A MT-912 Digital Light Meter (Urceri) was used to measure the background light intensity.
The CT dose meter phantom DICOM (digital imaging and communications in medicine) file was converted to an OBJ file (object file, Wavefront Technologies) and uploaded to the ARPPS for viewing on the HoloLens. The circumference and angle measurement tools of both the standard PACS and the ARPPS were used to measure diameters () and angles, respectively, with reference to the manufacturer-specified parameters of the CT dose meter phantom ( ).
A range of low, medium, and high clinical measurements were selected for anthropomorphic correlation of the phantom’s diameter and angle parameters (). Two readers measured each of the phantom parameters 10 times independently of each other starting with the ARPPS. The readers were blinded to the manufacturer-provided measurements. Testing was completed in an office with a background light intensity of 152.1 lux.
|Object||Manufacturer-specified size||Clinical guideline|
|Diameter A||3.215 cm||Mitral valve repair valve sizing  (mitral annulus diameter 3.15 cm)|
|Diameter B||5.0 cm||Elective abdominal aortic aneurysm repair in women  (5.0-5.4 cm)|
|Diameter C||21.31 cm||Pediatric abdominal diameter|
|Angle A||26.57°||Scoliosis evaluation  (bracing Cobb angle 29-40°)|
|Angle B||90.0°||Proximal tibial alignment  (normal lateral distal tibial angle 90°)|
|Angle C||153.43°||Pediatric hip evaluation  (normal pediatric femoral shaft angle 160°)|
All statistical analyses were performed using Microsoft Excel version 1903. The interrater reliability of the readers was verified using Lin’s concordance correlation coefficient for both the circumference and angle tools . Shapiro-Wilk test was performed to verify the normality of the differences of each set of measurements in order to satisfy the requirements of performing a nonparametric method of analysis such as a Bland-Altman analysis [ ]. Bland-Altman analysis was used to evaluate the agreement between measurements taken with the standard PACS and the ARPPS.
Lin’s concordance correlation coefficient showed almost perfect concordance of the standard PACS viewer and the ARPPS (, ). Additionally, no significant difference in interrater reliability was observed for the circumference and angle tool measurements for both the PACS and ARPPS separately ( , ).
The Shapiro-Wilk tests failed to reject the null hypothesis of normality (). Bland-Altman plots evaluating the circumference tool showed an average bias of 0.08% with a 95% CI –4.20% to 4.36%. Bland-Altman plots evaluating the angle tool showed an average bias of –1.84% with a 95% CI –6.17% to 2.14%. The bias and confidence intervals of each of the 3 measures for the circumference and angle tools are reported in . The Bland-Altman plots of each of the measurements, as well as the combined measurements are shown for the circumference tool ( a-d) and angle tool ( e-h).
The variability of the percent error of each of the measurements using the ARPPS as compared to using the standard PACS are visualized in individual box plots in.
|Tools||Concordance correlation coefficient||Interrater reliability PACSa standard DICOMb viewer||Interrater reliability ARPPSc|
aPACS: picture archiving and communication system.
bDICOM: digital imaging and communications in medicine.
cARPPS: augmented reality presurgical planning system.
|Tools and measurements||Shapiro-Wilk test |
|% Bias||Lower limits of agreement, %||Upper limits of agreement, %|
Principal Results and Comparison to Prior Work
Both the circumference and angle measuring tools of the ARPPS had an accuracy that was not significantly different as compared to the PACS measurements used in traditional preoperative settings. The circumference tool had an overall bias of 0.08%, which is more accurate than the 0.3% previously reported for a comparable AR system . Similarly, the angle tool had an overall bias of –1.84%, which is more accurate than that previously reported for another 3D reconstruction software already on the market [ ].
Interestingly, a decrease in percent error in either circumference or angle tool measurements was associated with an increase in the size of the object and ray length, respectively (). This was consistent with a corresponding increase in the ease of manipulation of the hologram for larger objects as reported by both readers. AR and mixed reality–viewing hardware with higher resolution and responsiveness is likely to significantly improve the usability of such systems.
Manipulating objects on the HoloLens can be technically challenging and contain a systematic error. Both readers reported difficulties in determining a clear vertex for angles A and C. However, angle B, which had no reported difficulties in measurement, showed a bias of only 0.14%. In addition, readers reported significant improvements in hologram manipulation dexterity with experience.
ARPPS can be an effective, precise, and accurate tool for the realistic visualization, manipulation, and measurement of clinically significant angles and circumferences in 3D space. ARPPS measurements are of substantially equivalent accuracy and precision as compared to standard-of-care PACS, similar systems that have previously been awarded the Food and Drug Administration (FDA) clearance as class II medical devices for presurgical planning, and other systems with published data [, ]. Nonetheless, technological difficulties remain a major barrier to the adoption of such technologies in medical and surgical care settings. To realize the full potential of AR and similar technologies, it is important that the medical community works in concert with device manufacturers to ensure the devices’ real-world feasibility, usability, safety, and efficacy.
We thank UCSF Benioff Children’s Hospital for the use of their CT and computers for obtaining PACS measurements. In addition, we thank the Microsoft HoloLens team for providing research and technology support for the HoloLens and Dr Nancy Hills, Associate Professor of Neurology, University of California, San Francisco for the advice on our biostatistics methods and analyses.
Conflicts of Interest
JC is an Associate Clinical Professor in Pediatric Radiology at the University of California, San Francisco and creator of the ARPPS but did not participate in the collection or analysis of the data.
- Choplin RH, Boehme JM, Maynard CD. Picture archiving and communication systems: an overview. RadioGraphics 1992 Jan;12(1):127-129. [CrossRef]
- Pilson HT, Reddix RN, Mutty CE, Webb LX. The long lost art of preoperative planning--resurrected? Orthopedics 2008 Dec;31(12). [CrossRef] [Medline]
- Shamshuddin S, Matthews H. Use of OsiriX in developing a digital radiology teaching library. Clin Radiol 2014 Oct;69(10):e373-e380. [CrossRef] [Medline]
- Khor WS, Baker B, Amin K, Chan A, Patel K, Wong J. Augmented and virtual reality in surgery-the digital surgical environment: applications, limitations and legal pitfalls. Ann Transl Med 2016 Dec;4(23):454 [FREE Full text] [CrossRef] [Medline]
- Dias CR, Pereira MR, Freire AP. Qualitative review of usability problems in health information systems for radiology. J Biomed Inform 2017 Dec;76:19-33 [FREE Full text] [CrossRef] [Medline]
- Thomas B, Demczuk V, Piekarski W, Hepworth D, Gunther B. A wearable computer system with augmented reality to support terrestrial navigation. : IEEE; 1998 Oct 19 Presented at: Second International Symposium on Wearable Computers; 1998; Pittsburgh, PA, USA. [CrossRef]
- Dunston PS, Wang X. Mixed Reality-Based Visualization Interfaces for Architecture, Engineering, and Construction Industry. J. Constr. Eng. Manage 2005 Dec;131(12):1301-1309. [CrossRef]
- Hořejší P. Augmented Reality System for Virtual Training of Parts Assembly. Procedia Engineering 2015;100:699-706. [CrossRef]
- Fjeld M, Voegtli BM. Augmented Chemistry: an interactive educational workbench. : IEEE; 2002 Presented at: International Symposium on Mixed and Augmented Reality; 2002; Darmstadt, Germany. [CrossRef]
- Ogdon DC. HoloLens and VIVE Pro: Virtual Reality Headsets. J Med Libr Assoc 2019 Jan 04;107(1):118-121. [CrossRef]
- Uppot RN, Laguna B, McCarthy CJ, De Novi G, Phelps A, Siegel E, et al. Implementing Virtual and Augmented Reality Tools for Radiology Education and Training, Communication, and Clinical Care. Radiology 2019 Jun;291(3):570-580. [CrossRef] [Medline]
- Komasawa N, Ohashi T, Take A, Doi Y, Kadoyama K, Terasaki F, et al. Hybrid simulation training utilizing augmented reality and simulator for interprofessional advanced life support training. J Clin Anesth 2019 Nov;57:106-107. [CrossRef] [Medline]
- Wellens LM, Meulstee J, van de Ven CP, Terwisscha van Scheltinga CEJ, Littooij AS, van den Heuvel-Eibrink MM, et al. Comparison of 3-Dimensional and Augmented Reality Kidney Models With Conventional Imaging Data in the Preoperative Assessment of Children With Wilms Tumors. JAMA Netw Open 2019 Apr 05;2(4):e192633 [FREE Full text] [CrossRef] [Medline]
- Qian L, Deguet A, Kazanzides P. ARssist: augmented reality on a head-mounted display for the first assistant in robotic surgery. Healthc Technol Lett 2018 Oct;5(5):194-200 [FREE Full text] [CrossRef] [Medline]
- Bourdel N, Chauvet P, Calvet L, Magnin B, Bartoli A, Canis M. Use of Augmented Reality in Gynecologic Surgery to Visualize Adenomyomas. J Minim Invasive Gynecol 2019;26(6):1177-1180. [CrossRef] [Medline]
- Low D, Lee CK, Dip LLT, Ng WH, Ang BT, Ng I. Augmented reality neurosurgical planning and navigation for surgical excision of parasagittal, falcine and convexity meningiomas. Br J Neurosurg 2010 Feb;24(1):69-74. [CrossRef] [Medline]
- Sato Y, Nakamoto M, Tamaki Y, Sasama T, Sakita I, Nakajima Y, et al. Image guidance of breast cancer surgery using 3-D ultrasound images and augmented reality visualization. IEEE Trans Med Imaging 1998 Oct;17(5):681-693. [CrossRef] [Medline]
- Boillat T, Grantcharov P, Rivas H. Increasing Completion Rate and Benefits of Checklists: Prospective Evaluation of Surgical Safety Checklists With Smart Glasses. JMIR Mhealth Uhealth 2019 Apr 29;7(4):e13447 [FREE Full text] [CrossRef] [Medline]
- Vávra P, Roman J, Zonča P, Ihnát P, Němec M, Kumar J, et al. Recent Development of Augmented Reality in Surgery: A Review. J Healthc Eng 2017;2017:4574172 [FREE Full text] [CrossRef] [Medline]
- Navab N, Blum T, Wang L, Okur A, Wendler T. First Deployments of Augmented Reality in Operating Rooms. Computer 2012 Jul;45(7):48-55. [CrossRef]
- Chang F, Laguna B, Uribe J, Vu L, Zapala MA, Devincent C, et al. Evaluating the Performance of Augmented Reality in Displaying Magnetic Resonance Imaging-Derived Three-Dimensional Holographic Models. J Med Imaging Radiat Sci 2020 Mar;51(1):95-102. [CrossRef] [Medline]
- Alletto C. 510(k) Premarket Notification K101342 - PIXMEO SARL OSIRIX MD. FDA 510(k) Premarket Notification Database. 2010. URL: https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPMN/pmn.cfm?ID=K101342 [accessed 2019-06-01]
- Dwivedi G, Mahadevan G, Jimenez D, Frenneaux M, Steeds RP. Reference values for mitral and tricuspid annular dimensions using two-dimensional echocardiography. Echo Res Pract 2014 Dec 01;1(2):43-50 [FREE Full text] [CrossRef] [Medline]
- Chaikof EL, Dalman RL, Eskandari MK, Jackson BM, Lee WA, Mansour MA, et al. The Society for Vascular Surgery practice guidelines on the care of patients with an abdominal aortic aneurysm. J Vasc Surg 2018 Jan;67(1):2-77.e2 [FREE Full text] [CrossRef] [Medline]
- Horne J, Flannery R, Usman S. Adolescent Idiopathic Scoliosis: Diagnosis and Management. Am Fam Physician 2014 Feb 01;89(3):193-198. [CrossRef]
- Chao E, Neluheni EV, Hsu RW, Paley D. Biomechanics of malalignment. Orthop Clin North Am 1994 Jul;25(3):379-386. [Medline]
- Soloman L. In: Warwick D, Nayagam S, editors. Apley's System of Orthopaedics and Fractures. Boca Raton, Florida: CRC Press; 2010.
- Lin LI. A concordance correlation coefficient to evaluate reproducibility. Biometrics 1989 Mar;45(1):255-268. [Medline]
- Altman DG, Bland JM. Measurement in Medicine: The Analysis of Method Comparison Studies. The Statistician 1983 Sep;32(3):307-317. [CrossRef]
- Merrill D. 510(k) Premarket Notification K172418 - Novarad Opensight. FDA 510(k) Premarket Notification Database. 2018. URL: https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K172418 [accessed 2019-06-01]
- Ilharreborde B, Steffen JS, Nectoux E, Vital JM, Mazda K, Skalli W, et al. Angle Measurement Reproducibility Using EOSThree-Dimensional Reconstructions in Adolescent Idiopathic Scoliosis Treated by Posterior Instrumentation. Spine 2011;36(20):E1306-E1313. [CrossRef]
|AR: augmented reality|
|ARPPS: augmented reality presurgical planning system|
|CT: computed tomography|
|DICOM: digital imaging and communications in medicine|
|FDA: Food and Drug Administration|
|MRI: magnetic resonance imaging|
|PACS: picture archiving and communication system|
|VR: virtual reality|
Edited by G Eysenbach, J Pearson; submitted 29.02.20; peer-reviewed by R Uppot, B Laguna, JA Sánchez-Margallo, D Koutsouris; accepted 13.08.20; published 24.09.20Copyright
©David Dallas-Orr, Yordan Penev, Robert Schultz, Jesse Courtier. Originally published in JMIR Perioperative Medicine (http://periop.jmir.org), 24.09.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Perioperative Medicine, is properly cited. The complete bibliographic information, a link to the original publication on http://periop.jmir.org, as well as this copyright and license information must be included.