This paper is in the following e-collection/theme issue:

Mobile tools for surgery and perioperative medicine (80) Quality Evaluation and Descriptive Analysis/Reviews of Multiple Existing Mobile Apps (371) Mobile Health (mhealth) (2893) mHealth in a Clinical Setting (607) Quality Improvement (245)

Published on in Vol 4, No 2 (2021): Jul-Dec

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/27037, first published .
Mobile Health Apps That Act as Surgical Preparatory Guides: App Store Search and Quality Evaluation

Mobile Health Apps That Act as Surgical Preparatory Guides: App Store Search and Quality Evaluation

Mobile Health Apps That Act as Surgical Preparatory Guides: App Store Search and Quality Evaluation

Authors of this article:

Naga Sindhura Gadde1 Author Orcid Image ;   Kevin Yi-Lwern Yap1, 2 Author Orcid Image
Article Authors Cited by (1) Tweetations (5) Metrics

    Original Paper

    1Department of Public Health, School of Psychology and Public Health, La Trobe University, Melbourne, Australia

    2Department of Pharmacy, Singapore General Hospital, Singapore, Singapore

    Corresponding Author:

    Kevin Yi-Lwern Yap, PhD

    Department of Pharmacy

    Singapore General Hospital

    SingHealth Tower, 10 Hospital Boulevard, Lobby A, Level 9

    Singapore, 168582

    Singapore

    Phone: 65 63214366

    Email: kevin.yap.y.l@sgh.com.sg


    Background: Mobile health (mHealth) apps are becoming increasingly common in surgical practices for training, education, and communication. Factors leading to increased delays, morbidity, and mortality in surgery include inadequate preoperative patient preparation due to a failure to identify patients and procedure details, and missing instruments and equipment required for the procedure. Many apps are available for supporting preoperative, intraoperative, and postoperative care. However, there is a lack of studies that assess the quality of apps that act as surgical preparatory guides.

    Objective: The aim of this study is to evaluate the quality of apps that act as surgical preparatory guides for operating room personnel through an in-house quality assessment tool.

    Methods: The quality assessment tool comprises 35 questions categorized into 5 sections: (1) engagement (customization, interactivity, target audience; 19 points), (2) functionality (performance, ease of use, navigation; 12 points), (3) aesthetics (layout, visual appeal; 6 points), (4) information (quality and quantity of information, visual information, credibility; 29 points), and (5) privacy and security (4 points). An app search was conducted in the Australian Apple and Google Play stores using the following keywords: “surgical apps”, “surgical preferences”, “surgeon preferences”, “operating room”, and “perioperative procedures”. The overall total scores and scores for each section were reported as medians and IQRs, expressed as raw scores and percentages.

    Results: A total of 5 unique apps were evaluated on both iOS and Android platforms. The median overall score across all apps was 35/70 (50%; IQR 38.6%-64.3%). ScrubUp (48/70, 69%) and MySurgeon (42/70, 60%) had the highest overall scores, followed by PrefCard (35/70, 50%) and Scrubnote (28/70, 40%). The lowest scoring app was BrainPadd (26/70, 37%). The sections with the highest median scores, in decreasing order, were privacy and security (4/4, 100%; IQR 75%-100%), aesthetics (5/6, 83%; IQR 75%-91.7%), engagement (15/19, 79%; IQR 57.9%-86.8%), functionality (7/12, 58%; IQR 29.2%-75%), and information (5/29, 17%; IQR 15.5%-34.5%). Most apps scored well (4/4, 100%) on privacy and security, except for Scrubnote (2/4, 50%). ScrubUp received a perfect score for aesthetics (6/6, 100%). MySurgeon (17/19, 90%) had the highest engagement score, while ScrubUp and MySurgeon had the highest functionality scores (9/12, 75% each). All apps scored below 50% for the information section, with ScrubUp having the highest score of 13/29 (45%).

    Conclusions: ScrubUp and MySurgeon had the highest quality scores and can be used as adjuncts to hospital protocols by operating room personnel for their surgical preparation. Developers are encouraged to develop appropriate apps for surgical preparation based on relevant guidelines and standards, as well as the quality evaluation criteria in our tool. Operating room personnel can also use this tool as a guide to select and assess their preferred apps in their practices.

    JMIR Perioper Med 2021;4(2):e27037

    doi:10.2196/27037

    Keywords

    mHealth apps; surgical apps; surgery preparation; operating room personnel; quality assessment; quality evaluation; perioperative; operative; mobile health; surgery; post-operative 


    As the global digital health market continues to flourish, technological innovations such as electronic medical record systems, laboratory and clinical information systems, mobile apps, health information technology, wearable devices, telehealth, and telemedicine are becoming more common in the health care setting to improve health service delivery and quality [1]. Mobile health (mHealth) technologies such as mobile phones and patient monitoring devices are becoming increasingly common in medical and surgical practices for training, education, and communication [1,2]. Surgical processes are complex, and a variety of factors may result in surgical delays and cancellations [3]. Major factors leading to increased delays, morbidity, and mortality in surgery include inadequate preoperative patient preparation resulting from a failure to identify patients and procedure details, and missing instruments and equipment required for the procedure [4]. Similarly, perioperative nurses face pressures to balance between maintaining operating room schedules and the surgeon’s demands due to time restrictions and the potential to underestimate the time required to operate [5]. Furthermore, newly graduated nurses who are uncertain about the intensified and unfamiliar situations during the initial stages of their clinical practice may face additional pressures in addressing emerging and complex technology, resulting in technological stress and surgical errors [6]; these pressures are in addition to trying to cope with the demands of new clinical placements and advancing their professional careers at the same time [7,8].

    Surgical preparation tools such as preassessment clinics and the World Health Organization Surgical Safety Checklist aim to minimize adverse events and errors in the operating room [3,9]. However, in some developing countries, surgical teams are still unable to use the safety checklists effectively [10]. With an increasing global trend of smartphone users to a predicted 3.8 billion users in 2021 [11], it is envisaged that smartphone apps may be useful to surgical teams, newly graduated health care professionals, or health care trainees to improve their awareness of and attitudes toward surgical safety practices since these apps can be accessed anytime and anywhere [10,12]. Some features of such apps that may enhance the usability of these safety checklists include the ability to customize surgical preparatory notes according to user preferences, provide comprehensive step-by-step information on preoperative and postoperative procedures, and provide clear and accurate visual explanations of the surgical procedures, for example, through images or videos. Furthermore, smartphone apps can act as quick reference guides that provide various clinical resources as a part of training and education for health care trainees, students, residents, fellows, and surgeons during their clinical practices to ultimately improve communication, system efficiency, and patient safety [1,13-15]. Studies have shown that many apps are available for supporting preoperative, intraoperative, and postoperative care [16], and that health care professionals working in surgical care and app developers are also producing innovative apps to aid surgical teams in education, training, and practice [15].

    Despite the widespread availability of mHealth apps, the literature has focused mostly on the prevalence and evaluation of communication, education, clinical, and diagnostic apps for physicians; health and medication monitoring apps for patients; and healthy living apps for diet, exercise, pregnancy, and heart rate monitoring for laypersons [13]. In the surgical domain, some studies have focused on the use of mobile patient health record apps and apps for diagnosis in the perioperative setting (eg, smartphone-based electrocardiograms, pulse oximetry, and blood glucose monitoring), while others have concentrated on the use of medical reference and perioperative crisis event management apps to improve care quality and safety in patient care, as well as on apps that facilitate patient monitoring and follow-up in the postoperative period [17]. The use of smartphones to promote better communication (eg, text messaging, emails) among the care team has also been studied [17]. However, there is a lack of studies that assess the quality of apps that act as surgical preparatory guides. Surgical preparatory guides help to support the surgical team in preparing the operating room and the patients before each procedure. They can consist of checklists of tools and equipment (eg, dressings, drapes, disposables, surgical instruments), information on preoperative safety procedures (eg, checking consent forms and diagnostic images, identification of patients and surgical sites), information about surgical preparatory steps (eg, putting up drapes, sterilizing surgical sites, preparing the patient and operating room, correct positioning of the patient), information on postoperative steps after each procedure (eg, recording correct counts of instruments, procedure names, specimens collected, equipment issues), and related reference sources, among others. Furthermore, there are concerns that there is no control on the quality of the apps, nor is there any regulated body that oversees the validity of the content unless the app is considered a medical device [18]. On the other hand, the lack of proper information on the quality of such apps and their content makes it difficult for users to identify the most useful apps, and there is also a risk that users may access misguided or misleading information [12,19]. Therefore, the objective of this study is to develop a quality assessment tool to evaluate apps that act as surgical preparatory guides for operating room personnel (nurses, surgical technicians, circulating nurses and technicians). The aim is to provide a recommendation of apps that may be useful to operating room personnel during their training and initial stages of clinical practice to improve patient safety and health communication within the surgical team.


    Development of the Quality Evaluation Tool

    The overall framework of the quality evaluation tool was adapted from the Mobile Application Rating Scale (MARS) [20], with modifications made to some criteria to fit the evaluation of apps for surgical preparation based on relevant articles found from a keyword search in PubMed (“surgical apps” OR “surgical applications” OR “ surgical safety” AND “quality tools” OR “quality scale” OR “assessment criteria” OR “evaluation”) [15,16,18,21-24]. In addition, we used relevant articles that reported quality assessment tools or criteria for evaluating general mHealth apps to refine the quality evaluation criteria of our tool [25-43].

    The tool was comprised of 35 questions categorized into 5 sections: (1) engagement (customization, interactivity, target audience; 19 points), (2) functionality (performance, ease of use, navigation; 12 points), (3) aesthetics (layout, visual appeal; 6 points), (4) information (quality and quantity of information, visual information, credibility; 29 points), and (5) privacy and security (security, privacy; 4 points) (Multimedia Appendix 1). The maximum possible score was 70 points.

    Selection of Apps

    We conducted an app search in August 2020 on the Australian Apple (iOS) and Android (Google Play) app stores with the keywords “surgical apps”, “surgical preferences”, “surgeon preferences”, “operating room”, and “perioperative procedures”, which resulted in an identification of 1110 apps (Figure 1). Among these apps, a total of 642 apps were not related to surgery and were excluded from screening. Among the 468 apps that were screened based on app name and description, a total of 458 apps were excluded based on the following exclusion criteria: education or examination-related, language other than English, and requiring a subscription or payment for access. Surgical-related apps that were not related to surgical preparatory guides were also excluded including communication or coordination tools, apps specific to a hospital or disease, apps targeting non–operating room personnel or patients, game-based apps, journal-related apps, and apps advocating for the purchase of surgical tools. There were 5 unique surgical apps that were evaluated on both iOS and Android platforms. The iOS and Android versions of the apps were evaluated on an iPhone 7 (Apple Inc) and Oppo A37F and X9079 (Guangdong Oppo Mobile Telecommunications Corp., Ltd) phones, respectively.

    ‎
    Figure 1. Flowchart summary of the app screening process.
    View this figure

    Evaluation and Data Analysis

    The apps were evaluated independently by 5 individual raters. Any discrepancies in scores were resolved by a discussion among all the raters, and the final scores were used for analysis. The iOS and Android versions for each unique app were not treated differently; the results were not reported in relation to the different platforms, but per unique app instead. We computed descriptive statistics for the overall total scores and the scores for each section of the apps using SPSS, version 26 (IBM Corporation) and reported these scores as medians and IQRs, expressed as the raw scores and percentages.


    The median overall score across all the apps was 35/70 (50%; IQR 38.6%-64.3%). The apps that had the highest overall scores were ScrubUp (48/70, 69%) and MySurgeon (42/70, 60%), while the lowest scoring app was BrainPadd (26/70, 37%). Among all the apps, only 3 (ScrubUp, MySurgeon, and PrefCard) had an evaluation score of 50% or above (Table 1). The sections with the highest scores, in decreasing order, were privacy and security (4/4, 100%), aesthetics (5/6, 83%), engagement (15/19, 79%), functionality (7/12, 58%), and information (5/29, 17%).

    In general, 4 apps took into consideration privacy and security aspects of their features, achieving the maximum score of 4/4 (100%). Only Scrubnote scored 2/4 (50%) as it did not explicitly state any privacy policy. All apps had a security login function for user authentication purposes.

    Table 1. Evaluation scores of the apps.

    		Evaluation scores for the surgical preparation apps, ranked from first (left) to last (right), points (%)Scores across all the apps


    		ScrubUpaMySurgeonbPrefCardcScrubnotedBrainPaddeMedian (%)IQR (%)
    Engagement (out of 19)16 (84)17 (90)15 (79)14 (74)8 (42)15 (79)11-16.5 (57.9-86.8)

    		Customization (out of 14)12 (86)13 (93)11 (79)10 (71)4 (29)11 (79)7-12.5 (50-89.3)

    		Interactivity (out of 4)3 (75)3 (75)3 (75)3 (75)3 (75)3 (75)3-3 (75-75)

    		Target audience (out of 1)1 (100)1 (100)1 (100)1 (100)1 (100)1 (100)1-1 (100-100)
    Functionality (out of 12)9 (75)9 (75)7 (58)3 (25)4 (33)7 (58)3.5-9 (29.2-75)

    		Performance (out of 2)1 (50)2 (100)2 (100)1 (50)1 (50)1 (50)1-2 (50-100)

    		Ease of use (out of 8)6 (75)6 (75)3 (38)1 (13)1 (13)3 (38)1-6 (12.5-75)

    		Navigation (out of 2)2 (100)1 (50)2 (100)1 (50)2 (100)2 (100) 1-2 (50-100)
    Aesthetics (out of 6)6 (100)5 (83)5 (83)4 (67)5 (83)5 (83)4.5-5.5 (75-91.7)

    		Layout (out of 4)4 (100)3 (75)3 (75)3 (75)3 (75)3 (75)3-3 (75-75)

    		Visual appeal (out of 2)2 (100)2 (100)2 (100)1 (50)2 (100)2 (100)1.5-2 (75-100)
    Information (out of 29)13 (45)7 (24)4 (14)5 (17)5 (17)5 (17)4.5-10 (15.5-34.5)

    		Quality and quantity of information (out of 12)4 (33)2 (17)1 (8)2 (17)1 (8)2 (17)1-3 (8.3-25.0)

    		Visual information (out of 9)4 (44)0 (0)0 (0)0 (0)0 (0)0 (0)0-2 (0-22.2)

    		Credibility (out of 8)5 (63)5 (63)3 (38)3 (38)4 (50)4 (50)3-5 (37.5-62.5)
    Privacy and security (out of 4)4 (100)4 (100)4 (100)2 (50)4 (100)4 (100)3-4 (75 – 100)

    		Privacy (out of 2)2 (100)2 (100)2 (100)0 (0)2 (100)2 (100)1-2 (50-100)

    		Security (out of 2)2 (100)2 (100)2 (100)2 (100)2 (100)2 (100)2-2 (100-100)
    Total score (out of 70)48 (69)42 (60)35 (50)28 (40)26 (37)35 (50)27-45 (38.6-64.3)

    aAllis Technology Pty Ltd.

    bMederi Services, LLC.

    cHeadjam Pty Ltd.

    dScrubnote LLC.

    eConnexxus LLC.

    Overall, all apps scored well in the aesthetics section (median 5/6, 83%; IQR 75%-91.7%). Only 1 app (ScrubUp) received a perfect score (6/6, 100%), while another app (Scrubnote) scored the lowest (4/6, 67%). In general, the majority of the apps scored well for visual appeal, with consistent colors and fonts, and a clear organization of the content on the screen interface. However, there were some minor difficulties in locating and selecting some icons on most apps, except ScrubUp; thus, these apps only scored 3/4 (75%) in terms of layout.

    The median engagement score for the apps was 15/19 (79%; IQR 57.9%-86.8%). MySurgeon (17/19, 90%) had the highest engagement score, followed by ScrubUp (16/19, 84%). BrainPadd scored the lowest in engagement (8/19, 42%). All the apps scored 3/4 (75%) for interactivity as they only allowed one method of feedback about the app. All of the content in the apps was appropriate for their target audiences (1/1, 100%). In terms of customization (median score 11/14, 79%), all apps allowed users to store personalized notes according to their own preferences and the preferences of their surgical team members. BrainPadd was the only app that did not allow users to edit any preloaded information about surgical preparatory procedures or tools, nor add any additional details or images to the preloaded information, thus scoring the lowest (4/14, 29%). There were only 2 apps that allowed limited customization of notifications (MySurgeon, PrefCard) and syncing of scheduled reminders or alerts (MySurgeon, ScrubUp).

    Functionality was the second lowest scoring section, with a median score of 7/12 (58%; IQR 29.2%-75%). While more than half of the apps (ScrubUp, PrefCard, BrainPadd) scored well due to their consistency in navigation (2/2, 100%), only 2 apps (MySurgeon, PrefCard) did not have any technical issues (2/2, 100% each). The top scoring apps in this section (ScrubUp and MySurgeon, 9/12, 75% each) also scored the highest for ease of use (6/8, 75% each). These apps enabled users to access saved information without the need for internet access and also provided useful help sections for navigating the apps. By contrast, Scrubnote (3/12, 25%) and BrainPadd (4/12, 33%) scored the lowest for this section.

    The information section was the lowest scoring section, with a median score of 5/29 (17%; IQR 15.5%-34.5%). All the apps scored below 50% in this section, with the highest scoring app (ScrubUp) having a score of only 13/29 (45%). PrefCard scored the lowest in this section (4/29, 14%). None of the apps had a preoperative surgical safety checklist or a checklist of postoperative steps to be completed after the surgical procedure. ScrubUp was the only app that had preloaded images of the surgical instruments/tools displayed to users, resulting in a score of 4/9 (44%) in terms of visual information. With regard to credibility, ScrubUp and MySurgeon had the highest scores (5/8, 63%), even though none of the apps provided any information on funding.


    Principal Findings

    This study analyzed 5 apps that act as surgical preparatory guides for operating room personnel who might need them in their practices, such as those in training or new to the operating room. Among them, 4 apps (ScrubUp, MySurgeon, Scrubnote, PrefCard) could be used by multiple surgical specialties, while BrainPadd was specific to plastic surgery. As surgery procedures are becoming more advanced and complex, the operating room environment needs to be coordinated efficiently through effective communication among the surgical team members [44]. Therefore, the evaluation criteria in the customization section of our developed tool were intended to support the communication of surgical team members in the operating room. Our tool also evaluated the engagement of the apps in terms of customization and interactivity, as well as the user-friendliness of the apps. In general, the engagement scores of all the apps were relatively high. All the apps allowed users to store personalized notes. Users were also able to provide feedback about the apps through at least one form of contact to the company or developer (eg, contact number, email, feedback form). In terms of user-friendliness, the top scoring apps, ScrubUp and MySurgeon, provided useful help sections for users to navigate the apps. Therefore, these apps could potentially be used by the surgical team with little extra training or resources. On the other hand, developers of the other evaluated apps could improve the functionality features, such as providing more detailed instructions or user guides on how to use the apps and providing more obvious navigation links between screens. Furthermore, the importance of checklists was highlighted in several studies as an effective communication tool that could impact operating room efficiency and reduce delays and errors in surgical settings [45-47]. The Surgical Safety Checklist by the World Health Organization was developed to address this challenge of minimizing common and avoidable risks in the operating room before, during, and after the surgery process [9]. Thus, the need for a surgical safety checklist was also evaluated as part of the information section when evaluating the surgical preparatory apps in this study.

    Providing evidence-based information is one of the important criteria for medical apps, and this can be said for surgical preparatory apps as well. Studies have shown that health care professionals are more inclined to use apps that can provide current and up-to-date information at the point of care in clinical practice [13]. Similarly, other studies have also reported that users value apps that can provide them with immediate access to information [48]. Our evaluation tool attempted to address these factors in the information section by assessing the up-to-dateness of the app content, as well as the presence of preloaded information and evidence-based references. Unfortunately, none of the apps evaluated in this study contained preoperative surgical safety or postoperative procedural checklists, nor did they provide references for their information. Even though ScrubUp scored the highest in the information section, it only contained checklists of the surgical instruments and tools needed and preloaded images. Interestingly, none of the apps had included any preloaded or linked videos to explain the surgical preparatory procedure. Video-based learning is a useful and effective way of learning about surgical preparation, especially among residents [49]. Although the quality of surgical videos on video sharing sites such as YouTube can be improved, this form of learning presents an opportunity and can be considered for inclusion by app developers, if the videos are accurate, reliable, and evidence-based [50].

    In our study, PrefCard was ranked third based on its overall evaluation score. We observed that certain features of the app could only be accessed by users who were on the app’s list of affiliated hospitals. As the raters in our study were not from the list of affiliated hospitals, app features such as customization, functionality, and quantity and quality of the content were evaluated based on the images and descriptions provided in the app stores and the developer website. Other criteria related to functionality, such as performance of the app (eg, technical bugs or crashes), having an autocomplete feature, and accessing saved information in offline mode might have been scored differently from another user who had full access to all the features of the app. Furthermore, during the evaluation of BrainPadd, there were some technical issues with regard to downloading and accessing the app at the later stages; hence, the raters evaluated some criteria based on the description on its website. As with PrefCard, users who have full access to BrainPadd might also score the app differently.

    Limitations

    A limitation of this study was that it only evaluated apps available in the English language and from the Australian app stores. Therefore, our results would need to be extrapolated with caution when applied to apps in other countries. In addition, this study only evaluated the app features that were free. There were some features in the apps that were available as in-app purchases, and these were not included in our evaluation. The evaluation criteria in our tool were developed specifically to assess apps that were meant to be surgical preparatory guides, and not all surgical apps as the variety of surgical apps was too broad. Thus, evaluators who want to use this tool to conduct their own evaluations of surgical apps would have to modify or adapt the criteria to fit their scope of evaluation. Moreover, usability and user acceptance studies or trials are beyond the scope of this study due to time limitations. Future studies should include evaluating the receptivity and acceptance of these apps among potential users, as well as involving appropriate health care professionals as evaluators, such as those in the surgical team. Lastly, this study did not take into account any updates to the apps after their evaluation in August 2020. Any updated versions of the apps might lead to different scores for the individual sections and the overall quality scores. Users should consider any new or updated features of the apps when interpreting our results.

    Conclusion

    This study, we developed a tool for evaluating apps that act as surgical preparatory guides. Based on our evaluation, ScrubUp and MySurgeon are among the apps with better scoring features and can be used as adjuncts to existing hospital protocols for surgery preparation. In addition, the evaluation criteria in our tool can provide a form of guidance for operating room personnel, surgical professionals, and trainees to evaluate their preferred apps in the future. Similarly, app developers are encouraged to develop apps that follow relevant guidelines and standards, as well as the quality criteria in this tool, so that better quality apps that are reliable and incorporate evidence-based content can be used for surgical practices. Where appropriate, we also encourage app developers to submit their apps to relevant regulatory agencies for further evaluation and feedback.

    Acknowledgments

    The authors would like to thank Dr Michael Liem, Mr Upendra Soppa, Mrs Jyothika Chunduru, and Mrs Vinila Kolla for their help in rating of the apps, and Mr Adamm Ferrier, Dr Ali Lakhani, and Mrs Jane Sheats for their advice on analysis and how this study relates to practice.

    Authors' Contributions

    KYLY conceived and designed the study. NSG conducted the evaluation of the apps and analyzed the results. NSG and KYLY wrote the manuscript. All authors agreed to the publication of the manuscript.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Quality evaluation tool developed for surgical preparation apps.

    DOCX File , 28 KB
    1. Mosa ASM, Yoo I, Sheets L. A systematic review of healthcare applications for smartphones. BMC Med Inform Decis Mak 2012 Jul 10;12:67 [FREE Full text] [CrossRef] [Medline]
    2. Ventola CL. Mobile devices and apps for health care professionals: uses and benefits. P T 2014 May;39(5):356-364 [FREE Full text] [Medline]
    3. Herrera-Usagre M, Santana V, Burgos-Pol R, Oliva JP, Sabater E, Rita-Acosta M, et al. Effect of a mobile app on preoperative patient preparation for major ambulatory surgery: protocol for a randomized controlled trial. JMIR Res Protoc 2019 Jan 16;8(1):e10938 [FREE Full text] [CrossRef] [Medline]
    4. Treadwell JR, Lucas S, Tsou AY. Surgical checklists: a systematic review of impacts and implementation. BMJ Qual Saf 2014 Apr;23(4):299-318 [FREE Full text] [CrossRef] [Medline]
    5. Bull RM, Fitzgerald M. The invisible nurse--behind the scenes in an Australian OR. AORN J 2004 Apr;79(4):810, 813-818, 821. [CrossRef] [Medline]
    6. Smith J, Palesy D. Technology stress in perioperative nursing: an ongoing concern. J Perioper Nurs 2018 Jun 01;31(2):3. [CrossRef]
    7. Duchscher JB. A process of becoming: The stages of new nursing graduate professional role transition. J Contin Educ Nurs 2008 Oct;39(10):441-50; quiz 451. [CrossRef] [Medline]
    8. Hezaveh MS, Rafii F, Seyedfatemi N. Novice nurses' experiences of unpreparedness at the beginning of the work. Glob J Health Sci 2013 Dec 27;6(1):215-222 [FREE Full text] [CrossRef] [Medline]
    9. Implementation Manual WHO Surgical Safety Checklist (First Edition). World Alliance for Patient Safety. 2008 May.   URL: https://www.who.int/patientsafety/safesurgery/tools_resources/SSSL_Manual_finalJun08.pdf [accessed 2021-11-15]
    10. Alvarez AG, Dal Sasso GTM, Iyengar MS. Mobile persuasive technology for the teaching and learning in surgical safety: content validation. Nurse Educ Today 2018 Dec;71:129-134. [CrossRef] [Medline]
    11. O'Dea S. Number of smartphone users worldwide from 2016-2021 (in billions). Statista. 2020.   URL: https://www.statista.com/statistics/330695/number-of-smartphone-users-worldwide/ [accessed 2021-01-07]
    12. Jusoh S. A survey on trend, opportunities and challenges of mHealth apps. Int J Interact Mob Technol 2017 Nov 27;11(6):73-85. [CrossRef]
    13. Baumgart DC. Smartphones in clinical practice, medical education, and research. Arch Intern Med 2011 Jul 25;171(14):1294-1296. [CrossRef] [Medline]
    14. Buijink AWG, Visser BJ, Marshall L. Medical apps for smartphones: lack of evidence undermines quality and safety. Evid Based Med 2013 Jun;18(3):90-92. [CrossRef] [Medline]
    15. Franko OI. Smartphone apps for orthopaedic surgeons. Clin Orthop Relat Res 2011 Jul;469(7):2042-2048 [FREE Full text] [CrossRef] [Medline]
    16. Warnock GL. The use of apps in surgery. Can J Surg 2012 Apr;55(2):77-78 [FREE Full text] [CrossRef] [Medline]
    17. Simpao AF, Lingappan AM, Ahumada LM, Rehman MA, Gálvez JA. Perioperative smartphone apps and devices for patient-centered care. J Med Syst 2015 Sep;39(9):102. [CrossRef] [Medline]
    18. O'Neill KM, Holmer H, Greenberg SLM, Meara JG. Applying surgical apps: smartphone and tablet apps prove useful in clinical practice. Bull Am Coll Surg 2013 Nov;98(11):10-18. [Medline]
    19. Boulos MNK, Brewer AC, Karimkhani C, Buller DB, Dellavalle RP. Mobile medical and health apps: state of the art, concerns, regulatory control and certification. Online J Public Health Inform 2014;5(3):229 [FREE Full text] [CrossRef] [Medline]
    20. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth uHealth 2015 Mar 11;3(1):e27 [FREE Full text] [CrossRef] [Medline]
    21. Bahadori S, Wainwright TW, Ahmed OH. Smartphone apps for total hip replacement and total knee replacement surgery patients: a systematic review. Disabil Rehabil 2020 Apr;42(7):983-988. [CrossRef] [Medline]
    22. Chernetsky Tejedor S, Sharma J, Lavallee DC, Lober WB, Evans HL. Identification of important features in mobile health applications for surgical site infection surveillance. Surg Infect (Larchmt) 2019 Oct;20(7):530-534. [CrossRef] [Medline]
    23. Weyers S, Van Calenbergh S, Van Nieuwenhove Y, Mestdagh G, Coppens M, Bosteels J. Quality assessment in surgery: where do we stand now and where should we be heading? Gynecol Surg 2014 Mar 21;11(2):89-95. [CrossRef]
    24. Zhang MWB, Ho RCM, Hawa R, Sockalingam S. Analysis of the information quality of bariatric surgery smartphone applications using the Silberg scale. Obes Surg 2016 Jan;26(1):163-168. [CrossRef] [Medline]
    25. Azad-Khaneghah P, Neubauer N, Miguel Cruz A, Liu L. Mobile health app usability and quality rating scales: a systematic review. Disabil Rehabil Assist Technol 2021 Oct;16(7):712-721. [CrossRef] [Medline]
    26. Boudreaux ED, Waring ME, Hayes RB, Sadasivam RS, Mullen S, Pagoto S. Evaluating and selecting mobile health apps: strategies for healthcare providers and healthcare organizations. Transl Behav Med 2014 Dec;4(4):363-371 [FREE Full text] [CrossRef] [Medline]
    27. Butcher R, MacKinnon M, Gadd K, LeBlanc-Duchin D. Development and examination of a rubric for evaluating point-of-care medical applications for mobile devices. Med Ref Serv Q 2015;34(1):75-87. [CrossRef] [Medline]
    28. Jin M, Kim J. Development and evaluation of an evaluation tool for healthcare smartphone applications. Telemed J E Health 2015 Oct;21(10):831-837. [CrossRef] [Medline]
    29. Lewis TL. A systematic self-certification model for mobile medical apps. J Med Internet Res 2013 Apr 24;15(4):e89 [FREE Full text] [CrossRef] [Medline]
    30. McGlynn EA. An evidence-based national quality measurement and reporting system. Med Care 2003 Jan;41(1 Suppl):I8-15. [CrossRef] [Medline]
    31. Nouri R, R Niakan Kalhori S, Ghazisaeedi M, Marchand G, Yasini M. Criteria for assessing the quality of mHealth apps: a systematic review. J Am Med Inform Assoc 2018 Aug 01;25(8):1089-1098 [FREE Full text] [CrossRef] [Medline]
    32. van Haasteren A, Gille F, Fadda M, Vayena E. Development of the mHealth app trustworthiness checklist. Digit Health 2019;5:2055207619886463 [FREE Full text] [CrossRef] [Medline]
    33. Wyatt JC. How can clinicians, specialty societies and others evaluate and improve the quality of apps for patient use? BMC Med 2018 Dec 03;16(1):225 [FREE Full text] [CrossRef] [Medline]
    34. Wyatt JC, Thimbleby H, Rastall P, Hoogewerf J, Wooldridge D, Williams J. What makes a good clinical app? Introducing the RCP Health Informatics Unit checklist. Clin Med (Lond) 2015 Dec;15(6):519-521 [FREE Full text] [CrossRef] [Medline]
    35. Ali EE, Teo AKS, Goh SXL, Chew L, Yap KY. MedAd-AppQ: a quality assessment tool for medication adherence apps on iOS and android platforms. Res Social Adm Pharm 2018 Dec;14(12):1125-1133. [CrossRef] [Medline]
    36. Anderson K, Burford O, Emmerton L. App Chronic Disease Checklist: protocol to evaluate mobile spps for chronic disease self-management. JMIR Res Protoc 2016 Nov 04;5(4):e204 [FREE Full text] [CrossRef] [Medline]
    37. Chen J, Cade JE, Allman-Farinelli M. The most popular smartphone apps for weight loss: a quality assessment. JMIR mHealth uHealth 2015 Dec 16;3(4):e104 [FREE Full text] [CrossRef] [Medline]
    38. Chyjek K, Farag S, Chen KT. Rating pregnancy wheel applications using the APPLICATIONS scoring system. Obstet Gynecol 2015 Jun;125(6):1478-1483. [CrossRef] [Medline]
    39. Guo Y, Bian J, Leavitt T, Vincent HK, Vander Zalm L, Teurlings TL, et al. Assessing the quality of mobile exercise apps based on the American College of Sports Medicine guidelines: a reliable and valid scoring instrument. J Med Internet Res 2017 Mar 07;19(3):e67 [FREE Full text] [CrossRef] [Medline]
    40. Loy JS, Ali EE, Yap KY. Quality assessment of medical apps that target medication-related problems. J Manag Care Spec Pharm 2016 Oct;22(10):1124-1140. [CrossRef] [Medline]
    41. McMillan B, Hickey E, Patel MG, Mitchell C. Quality assessment of a sample of mobile app-based health behavior change interventions using a tool based on the National Institute of Health and Care Excellence behavior change guidance. Patient Educ Couns 2016 Mar;99(3):429-435. [CrossRef] [Medline]
    42. Reynoldson C, Stones C, Allsop M, Gardner P, Bennett MI, Closs SJ, et al. Assessing the quality and usability of smartphone apps for pain self-management. Pain Med 2014 Jun;15(6):898-909. [CrossRef] [Medline]
    43. Van Singer M, Chatton A, Khazaal Y. Quality of smartphone apps related to panic disorder. Front Psychiatry 2015;6:96 [FREE Full text] [CrossRef] [Medline]
    44. Wolf FA, Way LW, Stewart L. The efficacy of medical team training: improved team performance and decreased operating room delays: a detailed analysis of 4863 cases. Ann Surg 2010 Sep;252(3):477-83; discussion 483. [CrossRef] [Medline]
    45. Pugel AE, Simianu VV, Flum DR, Patchen Dellinger E. Use of the surgical safety checklist to improve communication and reduce complications. J Infect Public Health 2015;8(3):219-225 [FREE Full text] [CrossRef] [Medline]
    46. Kilduff CLS, Leith TO, Drake TM, Fitzgerald JEF. Surgical safety checklist training: a national study of undergraduate medical and nursing student teaching, understanding and influencing factors. Postgrad Med J 2018 Mar;94(1109):143-150. [CrossRef] [Medline]
    47. Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AS, Dellinger EP, Safe Surgery Saves Lives Study Group. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med 2009 Jan 29;360(5):491-499. [CrossRef] [Medline]
    48. Mendiola MF, Kalnicki M, Lindenauer S. Valuable features in mobile health apps for patients and consumers: content analysis of apps and user ratings. JMIR mHealth uHealth 2015 May 13;3(2):e40 [FREE Full text] [CrossRef] [Medline]
    49. Mota P, Carvalho N, Carvalho-Dias E, João Costa M, Correia-Pinto J, Lima E. Video-based surgical learning: improving trainee education and preparation for surgery. J Surg Educ 2018;75(3):828-835. [CrossRef] [Medline]
    50. de'Angelis N, Gavriilidis P, Martínez-Pérez A, Genova P, Notarnicola M, Reitano E, et al. Educational value of surgical videos on YouTube: quality assessment of laparoscopic appendectomy videos by senior surgeons vs. novice trainees. World J Emerg Surg 2019;14:22 [FREE Full text] [CrossRef] [Medline]

    ‎
    MARS: Mobile App Rating Scale
    mHealth: mobile health

    Edited by R Lee; submitted 08.01.21; peer-reviewed by YT Cheung, M Ghazisaeedi, H Karim; comments to author 22.05.21; revised version received 30.05.21; accepted 25.10.21; published 30.11.21

    Copyright

    ©Naga Sindhura Gadde, Kevin Yi-Lwern Yap. Originally published in JMIR Perioperative Medicine (http://periop.jmir.org), 30.11.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Perioperative Medicine, is properly cited. The complete bibliographic information, a link to the original publication on http://periop.jmir.org, as well as this copyright and license information must be included.