Background: Mobile health (mHealth) apps are increasingly used postoperatively to monitor, educate, and rehabilitate. The usability of mHealth apps is critical to their implementation.
Objective: This systematic review evaluates the (1) methodology of usability analyses, (2) domains of usability being assessed, and (3) results of usability analyses.
Methods: The A Measurement Tool to Assess Systematic Reviews checklist was consulted. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting guideline was adhered to. Screening was undertaken by 2 independent reviewers. All included studies were assessed for risk of bias. Domains of usability were compared with the gold-standard mHealth App Usability Questionnaire (MAUQ).
Results: A total of 33 of 720 identified studies were included for data extraction. Of the 5 included randomized controlled trials (RCTs), usability was never the primary end point. Methodology of usability analyses included interview (10/33), self-created questionnaire (18/33), and validated questionnaire (9/33). Of the 3 domains of usability proposed in the MAUQ, satisfaction was assessed in 28 of the 33 studies, system information arrangement was assessed in 11 of the 33 studies, and usefulness was assessed in 18 of the 33 studies. Usability of mHealth apps was above industry average, with median System Usability Scale scores ranging from 76 to 95 out of 100.
Conclusions: Current analyses of mHealth app usability are substandard. RCTs are rare, and validated questionnaires are infrequently consulted. Of the 3 domains of usability, only satisfaction is regularly assessed. There is significant bias throughout the literature, particularly with regards to conflicts of interest. Future studies should adhere to the MAUQ to assess usability and improve the utility of mHealth apps.
Industry experts have forecasted significant growth in mobile app users . Given this projected surge, mobile health (mHealth) apps offer a unique and readily accessible platform to the patient, surgeon, and innovator. mHealth apps are now being integrated into various sectors of health care, with over 318,000 [ ] apps currently helping to track, educate, and diagnose [ ].
One area of particular growth is the use of mHealth apps as a means of monitoring patients in the important postoperative period. Well-designed apps have the potential to encourage earlier discharge, reduce in-person follow-ups [, ], rehabilitate [ ], aid clinicians in picking up surgical complications [ ], and improve communication between patient and health care professional [ ]. In addition to the economic and medical benefit of early discharge, postoperative monitoring apps have the potential to empower patients, giving them autonomy over their own health, which in turn might improve patient satisfaction and motivation for recovery [ ].
The usability of mHealth apps is important [, ] because those with poor usability will be less commonly used [ , ]. This is particularly significant in the postoperative period, given the focus of mHealth apps on rehabilitation, for which patient engagement is critical. One study revealed that around half of all mHealth app users stop engaging for various reasons, including loss of interest [ ]. Despite this, little empirical research is undertaken to analyze the usability of mHealth apps before they are launched [ ].
Several definitions and domains of usability have been previously defined without clear unification [, , ], but with several recurring themes. For example, the International Organization for Standardization (ISO) 3-pronged definition includes effectiveness (ie, whether users can use the product to complete their goals), efficiency (ie, the extent to which individuals expend resource in achieving their goals), and satisfaction [ ]. Another definition [ ] has been designed specifically for mHealth apps and includes factors such as mobility, connectivity, and additional cognitive load.
Different methods have been proposed for assessing domains of usability, such as the Post-Study System Usability Questionnaire  and the System Usability Scale (SUS) [ ]. However, these tools were not originally created to evaluate mHealth apps. The Mobile App Rating Scale [ ] was recently created for researchers and clinicians to assess the quality of mHealth apps, with the simpler user version of the Mobile App Rating Scale (uMARS) [ ] being proposed shortly after. While quality of an mHealth app shares several components with usability, there are important differences.
Given the heterogeneity in definitions and methods used for assessing the usability of mHealth apps, one group has recently developed and validated the 21-item mHealth App Usability Questionnaire (MAUQ) . This tool explores 3 domains of usability, which are in line with the ISO definition: (1) ease of use and satisfaction, akin to ISO satisfaction; (2) system information arrangement, akin to ISO efficiency; and (3) usefulness, akin to ISO effectiveness. This systematic literature review aims to determine whether the usability of postoperative mHealth apps is being rigorously assessed, using the validated MAUQ as the gold-standard reference. We consider which empirical methods are being used and analyze whether postoperative mHealth apps are indeed usable.
The A Measurement Tool to Assess Systematic Reviews checklist  was analyzed before conducting this review, with all methodology being established prior to the review being conducted. A university librarian experienced in the field of systematic literature review methodology was consulted. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ ] reporting guideline was adhered to for this review. Rayyan (Qatar Computing Research Institute) [ ] software was used for the search.
shows the questions that were defined.
The Medline, Embase, and Association for Computing Machinery Digital Library databases were searched. The search string was generated and aimed to provide maximum coverage while maintaining manageability. We defined 4 broad themes for our search. Terms within a theme were combined using Boolean operator OR, as seen in. Themes were then combined using Boolean operator AND.
1. Which dimensions of usability are dealt with most often?
2. Which empirical methods are used to evaluate usability?
3. In which surgical specialties are mobile health apps’ usability being evaluated?
4. What types of operating systems have been used?
5. What are the results obtained by the usability evaluation of the apps?
|Mobile context||Smartphone OR smart phone OR mobile phone OR mobile device OR mHealtha OR tablet|
|Software||App OR application OR operating system OR OSb OR ios OR android OR windows OR google play|
|Postoperative||Postoperative OR post-operative OR surgery OR surgical OR operation OR perioperative OR peri operative|
|Usability||Usab* OR understandab* OR learnab* OR operab* OR attractive* OR user experience OR engag* OR satisf* OR adher* OR willing* OR accepta* OR effectiv* OR aesthetic OR intuitive*|
amHealth: mobile health.
bOS: operating system.
Screening of Papers for Inclusion and Exclusion
Each study recruited from the initial search was evaluated to determine whether it should be admitted for analysis. The inclusion and exclusion criteria are shown in.
Screening of article titles and abstracts was performed by 2 authors independently. In situations where eligibility of a study could not be determined based on abstract alone, the full-text article was retrieved. We executed a full-text review of the remaining studies after title and abstract screening to further analyze appropriateness for inclusion. We analyzed all review articles to identify any other appropriate studies. We also reviewed the reference list of included papers.
Inclusion and exclusion criteria.
- The paper uses a mobile health app, defined as an application (rather than a web-based tool) on a portable device (including smartphones and tablets). We include apps designed both for the patient and for the health care professional. We include all types of apps, including monitoring, educational, and rehabilitation apps
- The paper analyzes the postoperative period, defined as the point at which the patient leaves the operating theater, having undergone a surgical procedure
- The paper studies usability of the mobile health app. Any level of assessment is included, from structured questionnaire to analysis of engagement or time spent on the app
- The paper must be a full paper (not an abstract)
- The paper is not written in English
- The paper was published before 2000, in keeping with the launch of the first smartphone, the Ericsson R380 (Ericsson Mobile Communications)
- The paper only uses web-based, text-based, or email-based technologies (no mobile health app). We want to concentrate on mobile health apps, given that they are the subject of such traction in the market
- The app is not targeted to the postoperative period. For example, surgical apps monitoring patients following trauma or burns are excluded if no operative intervention is used. Furthermore, nonsurgical papers (eg, monitoring patients with chronic pain) are excluded. In addition, apps only used for education of surgeons are excluded
- Inappropriate study types, including reviews, case reports, and feasibility/pilot studies without any real-life postoperative analysis
- App is not designed for humans
Database Search Results
The initial search and reference list screening identified 721 studies. After title and abstract screening, 660 were excluded, leaving 61 full-text studies to be assessed. Of these, 28 were excluded, leaving 33 studies included for data extraction. The PRISMA summary of the database search is presented in.
A total of 33 studies were included. Of the 33 studies, 21 were from North America (14 from the United States and 6 from Canada), 9 were from Europe, 2 were from Asia, and 1 was from South America. Most studies specified the type of mobile device used by participants. Smartphones were used in 22 studies, tablets were used in 9, smartwatches were used in 1, iPod touch (Apple Inc) devices were used in 2, and 3 studies did not specify. Regarding the operating system, 11 studies used iOS (Apple Inc), 5 used Android, 1 used Windows (Microsoft Corp), and 17 did not specify.
Among the included studies, mHealth apps were used within a wide range of surgical subspecialties, including orthopedics (8 studies), general surgery (6 studies), head and neck (4 studies), transplant (3 studies), pediatrics (2 studies), breast (1 study), vascular (1 study), neurosurgery (1 study), and others/multiple (7 studies).
Functionality was divided into 5 clear categories; 26 studies included monitoring of symptoms or wounds, 8 included educational content, 5 provided a communication platform, 5 included physiotherapy and rehabilitation, and 2 enabled medication management. App details are presented in.
Study characteristics are presented in. With regards to study design, 5 studies were randomized controlled trials (RCTs), 25 were prospective noncontrolled studies, and 3 were retrospective reviews. Sample sizes ranged from 4 to 494, with a median of 39 patients and a mean of 81 patients. Follow-up ranged from 30 minutes postoperation to 12 months postdischarge. The follow-up period was less than 7 days in 4 studies, between 1 week and 1 month in 15 studies, greater than 1 month in 9 studies, and not declared in 5 studies.
|Study||Country||Primary mobile device (operating system)||App name||Surgical subspecialty||Function|
|Timmers et al ||Netherlands||Smartphone and tablet (—a)||Patient Journey App (Interactive Studios)||Orthopedics (elective total knee replacement)||Personalized educational information regarding pain; Physiotherapy; Wound monitoring; Self-care|
|Yadav et al ||India||Smartphone (—)||WhatsApp (Facebook Inc)||Endocrine surgery||Tele–follow-up including wound check and communication|
|Ramkumar et al ||United States||Smartphone (iOS)||TKR (Focus Ventures)||Orthopedics (elective total knee replacement)||Monitoring of mobility and range of movement using wearable sleeve; PROMsb; Analgesia need; Home exercise program compliance|
|Argent et al ||Ireland||Tablet (Android)||—||Orthopedics (elective total knee replacement)||Rehabilitation using an inertial measurement unitc, consisting of wearable sleeve; PROMs monitoring, including pain and perceived exercise difficulty|
|Brunner et al ||United States||Tablet (iOS)||Proloquo2Go (AssistiveWare)||Head and neck surgery||Augmentative and alternative communication in patients who are unable to speak postoperatively|
|van der Meij et al ||Netherlands||Smartphone (—)||—||Abdominal surgery (laparoscopic cholecystectomy, inguinal hernia surgery, laparoscopic adnexal surgery)||Information about surgical procedure; Insight into convalescence plan; Recovery monitor|
|Felbaum et al ||United States||Smartphone (—)||TrackMyRecovery||Neurosurgery||Postoperative instructions; Pain reporting; Wound monitoring|
|Goz et al ||United States||Smartphone (—)||—||Spine surgery||Postoperative communication through messaging app|
|Gunter et al ||United States||Smartphone (iOS)||WoundCheck||Vascular surgery||Wound monitoring using photographs and questionnaire|
|Gustavell et al ||Sweden||Smartphone and tablet (—)||Interaktor (Health Navigator)||Pancreatic surgery||Symptom monitoring; Education links to evidence-based care advice|
|Harder et al ||United Kingdom||Smartphone (iOS)||bWell||Breast surgery||Rehabilitation (arm exercises); Symptom monitoring|
|Higgins et al ||Canada||Smartphone (—)||QoC Health (QoC Health Inc)||Orthopedics (ACLd reconstruction)||Symptom monitoring; QoR-9e questionnaire|
|Highland et al ||United States||Smartphone (—)||mCare||Surgery using peripheral nerve block||Symptom control using DVPRSf|
|Khanwalkar et al ||United States||Smartphone (—)||HealthLoop||ENTg (septoplasty and FESSh)||PROMs, including VASi pain score, PROMISj, and SNOT-22k|
|Mata et al ||Canada||Tablet (iOS)||SeamlessMD (Seamless Mobile Health Inc)||Colorectal surgery||Milestones checklist; Symptom-monitoring questionnaires; Educational content|
|Nilsson et al ||Sweden||Smartphone (—)||Recovery Assessment by Phone Points||Day surgery||SwQoRl questionnaire|
|Pecorelli et al ||Canada||Smartphone (—)||SeamlessMD||Colorectal surgery||Milestones checklist; Symptom-monitoring questionnaires; Educational content|
|Sousa and Turrini ||Brazil||Smartphone (iOS)||OrtogApp||Orthognathic surgery||Educational content; Communication platform|
|Sun et al ||Canada||iPod touch (iOS)||Panda (Balsamiq Solutions)||Pediatric surgery||Postoperative pain monitoring; Medication management|
|Tsapepas et al ||United States||Tablet (—)||Medication Regimen Education||Kidney transplant||Educational content|
|Scott et al ||United States||Smartphone (—)||SeamlessMD||Colorectal surgery||Symptom tracker; Photograph of wound; Temperature recording|
|Warren-Stomberg et al ||Sweden||Smartphone (iOS and Android)||Medipal (Novatelligence AB)||Day orthopedic surgery||Symptom questionnaire|
|Debono et al ||France||Smartphone and tablet (—)||—||Lumbar discectomy||Symptom monitoring|
|Gunter et al ||United States||— (iOS)||WoundCheck||Vascular and general surgery||Symptom monitoring; Photograph of wound|
|Ponce et al ||United States||— (iOS)||HelpLightning||Orthopedics and neurosurgery||Virtual examination|
|Jiang et al ||United States||Smartphone (Windows)||PocketPATH||Lung transplant||Data entry of health indicators; Self-monitoring|
|Chai et al ||South Korea||Tablet (iOS)||Self-Reporting Application||Thyroid surgery||Self-reporting of symptoms|
|Shellmer et al ||United States||— (Android)||Teen Pocket PATH||Solid organ transplant||Monitoring of medications|
|Sun et al ||Canada||Smartphone (—)||Panda||Pediatric surgery||Postoperative pain monitoring using electronic versions of FPS-Rm and CASn|
|Jaensson et al ||Sweden||Smartphone (—)||Recovery Assessment by Phone Points||Day surgery||SwQoR questionnaire|
|Symer et al ||United States||Smartphone (iOS and Android) with paired smartwatcho||—||Colorectal surgery||Pain monitoring; Symptom monitoring; Patient reminders/alerts; Photograph of wound|
|Semple et al ||Canada||Smartphone or tablet (Android)||QoC Health||Breast and orthopedic surgery||Mobile version of the QoR-9 questionnaire|
|Bini and Mahajan ||United States||iPod touch (iOS)||CaptureProof||Orthopedic surgery||Physiotherapy videos|
bPROMs: patient-reported outcome measures.
dACL: anterior cruciate ligament.
eQOR-9: quality of recovery 9.
fDVPRS: Defense and Veterans Pain Rating Scale.
gENT: ear, nose, and throat.
hFESS: functional endoscopic sinus surgery.
iVAS: visual analog scale.
jPROMIS: Patient-Reported Outcomes Measurement Information System.
kSNOT-22: Sino-Nasal Outcome Test 22.
lSwQoR: Swedish Web Version of Quality of Life.
mFPS-R: Faces Pain Scale – Revised.
nCAS: color analog scale.
oFitbit; Fitbit Inc.
|Study||Study design||Number of patients||Duration follow-up||Method of analysis of usability /outcome measure||Aspects of usability measured||Selected quantitative measure of usability|
|Timmers et al ||Multicenter RCTa||213||4 weeks||Measurement of patient usage; Interview of small group of patients (n=6)||Usefulness||App used 26 times/patient; Videos watched 36 times/patient; Qualitative reporting of usefulness|
|Yadav et al ||Prospective study (no control)||107||6 months||Self-created questionnaire||Satisfaction; Usefulness||1% unsatisfied across the questionnaire; 53% very satisfied with effectiveness; 78% very satisfied with app overall; Comfortable: 78% very satisfied; Convenience: 86%-91% very satisfied|
|Ramkumar et al ||Prospective study (no control)||22||3 months||Semi-structured interview||Satisfaction; Usefulness||A1: average score 2.6/10 (1=easiest to use; 10=most difficult)|
|Argent et al ||Mixed methods, including prospective study||15||2 weeks||Questionnaires (SUSb and uMARSc); Semi-structured interview||Satisfaction; System information arrangement; Usefulness||uMARS average score 4.1/5 (SD 0.39); SUS average score 90.8 (SD 7.8)|
|Brunner et al ||Prospective preintervention and postintervention study||38||4 days||Self-created questionnaires; Measurement of usage||Satisfaction; Usefulness||66% used the app; 60% satisfied with the app; 85% felt it was helpful|
|van der Meij et al ||RCT||344||3 months||Measurement of usage; Self-created questionnaire; Semistructured interviews||Satisfaction||49.6% had used the app; Mean score for app 7.6/10|
|Felbaum et al ||Prospective study (no control)||56||—d||Self-created questionnaire||Usefulness||Usefulness ranged from 8.39-9.0 out of 10 (Likert scale)|
|Goz et al ||Prospective study (no control)||21||2 weeks||Measurement of usage/engagement; Self-created questionnaire||Satisfaction; Usefulness||82% satisfied (would recommend to others); 75% found useful (felt the app made it less likely for them to call the clinic); Engagement: 3.38 messages/person over 2 weeks|
|Gunter et al ||Prospective study (no control)||40||2 weeks||SUS (questionnaire); Measurement of usage||Satisfaction; System information arrangement||SUS average score of 87.2|
|Gustavell et al ||Prospective study (no control)||6||4 weeks||Measurement of usage; Semistructured interviews||Satisfaction; System information arrangement; Usefulness||Adherence to reporting daily was 84%; Other measurements qualitative|
|Harder et al ||Prospective study (no control)||4||8 weeks||Measurement of usage; Self-created questionnaire||Satisfaction; System information arrangement; Usefulness||Overall rating (Likert scale) 4.6/5; All used the app almost daily or several times/day|
|Higgins et al ||Retrospective case series||32||6 weeks||Interview; Self-created questionnaire||Satisfaction||Overall satisfaction was reported as excellent (43%), good (40%), fair (10%), poor (7%); 94% would use the app again|
|Highland et al ||RCT||24 (only 12 assessed usability)||10 days||SUS questionnaire; Additional questionnaire||Satisfaction; System information arrangement; Usefulness||SUS average score 76.26/100; No difference in convenience between intervention and standard of care (telephone follow-up)|
|Khanwalkar et al ||Prospective study (no control)||249||3 months||Measurement of usage||None||77.4% response rate (usage)|
|Mata et al ||RCT||50||4 weeks; Satisfaction measured at discharge||Measurement of usage; Self-created questionnaire using 4 items from S-CAHPSe||Satisfaction||Usage: postoperative day 0=94%, day 1=82%, day 2=72%, day 3=48%; 4/5 satisfaction across all 4 questions|
|Nilsson et al ||Prospective study (no control)||494||14 days||Measurement of usage (response rate)||None||Usage: day 1=86.8%, day 7=69%, day 14=57.5%|
|Pecorelli et al ||Prospective study (no control)||45||4 weeks||SUS questionnaire||Satisfaction; System information arrangement||SUS average score 87/100|
|Sousa and Turrini ||Prospective study (no control)||30||—||SUS questionnaire; Satisfaction measured according to experience sampling method technique; Usage||Satisfaction; System information arrangement||SUS average score 79.8/100, 73.3% >68 (cutoff), 100% >50 (acceptable); Satisfaction 82.9%; Usage: 100% used at least once, 40% used 2-3 times, 10% used 5 times, 20% used >5 times|
|Sun et al ||Prospective study (no control)||29||—||CSUQf Unstructured interviews||Satisfaction; System information arrangement; Usefulness||Median CSUQ score 2 (IQRg 1-3); 93% found app easy to use; 59% would use the app at home|
|Tsapepas et al ||Retrospective study||282||—||Self-created questionnaire||Satisfaction||Satisfaction rated 4 or 5 in 92%|
|Scott et al ||Prospective study (no control)||20||14 days||SUS questionnaire; Semi-structured interview; Measurement of usage||Satisfaction; System information arrangement; Usefulness||Median SUS 95/100; Usage: 30% did not use after discharge|
|Warren-Stomberg et al ||Prospective study (no control)||101||1 week||Measurement of usage||None||55/101 used the app; Of those that used the app, 53% used >13 times out of possible 15|
|Debono et al ||Prospective study (no control)||60||15 days||Telephone interview||Satisfaction; Usefulness||1 (poor) to 4 (excellent) scale: Overall satisfaction 3.4 Usability 3.5 Usefulness at home 3.2 Facilitating return at home 3.1; 91.6% would use the device again|
|Gunter et al ||Prospective study (no control)||9||—||SUS questionnaire||Satisfaction; System information arrangement||Average SUS score 83.3/100; 55.6% were able to complete the tasks independently|
|Ponce et al ||Prospective||31||24 days||15-point questionnaire||Satisfaction; Usefulness||Reassurance 4.6-4.8/5; Useful 4.5-4.8/5; Satisfaction 4.2-4.6/5|
|Jiang et al ||Secondary retrospective analysis of previous RCT data||96||12 months||Technology acceptance subscales used to measure: intention to use (1 item); perceived usefulness (4 items); and perceived ease of use (4 items)||Satisfaction; Usefulness||85% strongly agree with intention to use item; 80% gave high rating of perceived usefulness (>24/28); 82% gave high rating of perceived ease of use (>24/28)|
|Chai et al ||Prospective comparison study (nonrandomized)||54||14 days||Self-created questionnaire||Satisfaction; Usefulness||Satisfaction was >7.2/10 across all 4 items on questionnaire|
|Shellmer et al ||Prospective study||7||6 weeks||8/16 questions from PSSUQh survey||Satisfaction; System information arrangement; Usefulness||Satisfaction 1/7 (1=strongly agree); Ease of use 1/7; Felt comfortable using application 1/7; “I could clearly tell when I missed my medication” 1/7; Liked tracking medications 3/7; Helpful to track medications 2/7|
|Sun et al ||Prospective study||66||30 minutes postoperation||Single question asked regarding preference of monitoring (app vs paper version of questionnaire)||Satisfaction||76%-81% preferred the app over the paper version|
|Jaensson et al ||Prospective study||10||—||Self-created questionnaire on system layout and technical issues, satisfaction, and usefulness||Satisfaction; System information arrangement; Usefulness||—|
|Symer et al ||Prospective study||31||30 days||Measurement of usage; Self-created questionnaire||Satisfaction; System information arrangement; Usefulness||83.9% used the app 70% of the time; 89.3%: easy to navigate; 88.9%: easy to use; 85.2%: survey questions relevant for identifying problems related to readmission; 66.7% found reminders useful; 92.9% would recommend to others|
|Semple et al ||Prospective study||65||30 days||Self-created survey; Interview; Usage||Satisfaction||Satisfaction 3.7-3.9/4; 100% wiling to use in future; 100% surgeons found platform intuitive and easy to use; Usage: mean number of logins 19.3-23.9/30 days; Mean number of photographs uploaded 38-63/30 days|
|Bini and Mahajan ||RCT||29||24 weeks||Self-created survey; Free-form feedback; Usage||Satisfaction||Ease of use: 3.9-4.4/5; Satisfaction 4.2/5|
aRCT: randomized controlled trial.
bSUS: System Usability Scale.
cuMARS: user version of the Mobile App Rating Scale.
eS-CAHPS: Surgical Care Consumer Assessment of Healthcare Providers and Systems.
fCSUQ: Computer System Usability Questionnaire.
gIQR: interquartile range.
hPSSUQ: Post-Study System Usability Questionnaire.
Regarding the method of usability analysis, usage (ie, monitoring of user engagement with the app) was used in 15 studies and was the only usability analysis employed in 4 studies. Interviews were used in 10 studies. Self-created questionnaires were used in 18 studies. Validated questionnaires were used in 9 studies. Of these, 7 used the SUS questionnaire, 1 used the uMARS questionnaire, 1 used the technology acceptance subscale, and 1 used the Computer System Usability Questionnaire (CSUQ).
We have categorized the domains of usability according to the MAUQ. A total of 28 studies covered ease of use and satisfaction, 11 studies covered system information arrangement, and 18 studies covered usefulness.
Average SUS scores ranged from 76 to 95 out of 100, with a median score of 87. The uMARS score was 4.1 out of 5. The CSUQ score was 2 out of 7 (whereby a score of 1 would indicate greatest usability).
There is significant potential for bias in studies evaluating the usability of mHealth apps. Hidden agenda bias and secondary gains bias were common and seemingly underreported in the literature. Of the 33 included studies, 8 officially reported authors’ conflicts of interest, stating that they held shares in the app. Furthermore, several of the study groups were provided with the apps free of charge , which has clear implications on the usability domain of satisfaction; users who have paid for an app might be expected to have higher expectations than those who have been given an app for free. Perhaps more worryingly, a number of groups [ ] declared no conflict of interest, despite seemingly being founders of their app.
Nonresponse bias is a further concern. Some studies, such as Pecorelli et al , had high response rates (96%) to usability analyses. However, others, such as Nilsson et al [ ], had much lower rates (57.5% on day 14), and some [ ] did not disclose the proportion of responders. Nonresponders to usability analyses are more likely to have reported poor usability. Therefore, studies with high rates of nonresponders are likely to have inflated usability results.
Population bias is a further issue. Younger audiences are likely to be more adept at using mobile technologies. Therefore, studies that include a younger demographic are likely to demonstrate inflated usability results. Additionally, the generalizability of results from studies  that included patients that were not used to mobile technologies may be limited and may change in the future, when greater numbers of older patients are used to mobile technologies.
To our knowledge, this is the first comprehensive systematic review to assess usability of mHealth apps in postoperative management. This review identified 33 studies evaluating the usability of mHealth apps in the postoperative period across a broad range of surgical subspecialties, demonstrating the growing interest in this area. Most of the included studies were derived from the United States and Europe, which appear to be hubs of innovation in the field. Unsurprisingly, smartphones were the most commonly used devices. However, we suspect that wearable devices such as smartwatches, which have additional monitoring capabilities such as electrocardiogram monitors, will play an increasingly important role in the future .
With respect to study designs, 25 of 33 studies were prospective noncontrolled trials. There were 5 RCTs, but usability was never a primary end point in these studies. We feel RCTs comparing mHealth apps to normal practice (eg, in-person follow-up, telephone follow-up, or no follow-up) would be particularly beneficial in assessing the domains of satisfaction and usefulness. It has also been suggested that mHealth app interventions are associated with a falsely heightened level of user satisfaction due to patients’ affinities for their digital devices . This could be minimized by comparing postoperative mHealth apps to a sham app. However, we also acknowledge that RCTs have previously been described as an impractical evaluation methodology for mHealth apps, due to their prolonged duration from recruitment to results and their high costs [ ].
The methodology for assessing usability was generally poor. The majority of analyses used simplistic self-created questionnaires that asked rudimentary questions focusing on the domain of satisfaction (28/33 studies) rather than other domains of usability. Indeed, only 11 of the 33 usability analyses assessed the domain of system information arrangement. We would argue that formal usability analyses should cover all 3 common domains of (1) satisfaction, (2) usefulness, and (3) system arrangement, according to the ISO definition of usability . Validated questionnaires are helpful in assessing these areas reliably. Only 9 of the 33 included studies used validated questionnaires, most of which used the SUS. The SUS is a Likert scale made up of 10 questions. The average SUS score is 68 out of 100, meaning that all 7 studies that used the SUS scored above average in terms of usability. Although the SUS is a quick and cheap means of assessing usability, it was created in 1986, before the first smartphone or the concept of an app was realized. The SUS has not been validated for assessing mHealth apps. In comparison, the MAUQ was recently proposed and validated for use in mHealth apps in a population of English-speaking adults [ ]. This is the gold-standard reference for analysis of mHealth app usability. While scores on the MAUQ have previously been shown to correlate with the SUS, this is not a strong correlation (r=0.643), thereby highlighting the inadequacy of studies that have only used the SUS.
A major concern in these studies is the risk of bias. A number of the studies’ authors have a financial interest in the usability of their apps, with high user satisfaction making adoption by hospitals and investors more likely. Furthermore, devices were sometimes provided free of charge, which could influence the feedback from users.
mHealth apps have significant potential during the postoperative period for encouraging earlier discharge, improving patient engagement, and offering a safety net for early identification of complications. Thorough analysis of usability is critical to the adoption of these novel technologies in the postoperative period; those with poor usability will have little impact in health care. According to this review, usability analyses to date have been substandard. They have focused on satisfaction, a narrow dimension of usability, with simplistic self-created questionnaires. Furthermore, there is a significant risk of bias, given the common conflicts of interest among authors of published studies. We hope this review changes future practice, with researchers undertaking more robust assessments of usability by employing validated questionnaires, such as the MAUQ, in blinded RCTs.
BP contributed the study conception and design. BP and AT performed the acquisition of data and analysis of data. BP drafted the manuscript. All work was self-funded.
Conflicts of Interest
- Ericsson Mobility Report. Ericsson website. URL: https://www.ericsson.com/en/mobility-report [accessed 2018-05-29]
- The Growing Value of Digital Health. IQVIA Institute. 2017 Nov 07. URL: https://www.iqvia.com/institute/reports/the-growing-value-of-digital-health [accessed 2018-05-29]
- Byambasuren O, Sanders S, Beller E, Glasziou P. Prescribable mHealth apps identified from an overview of systematic reviews. NPJ Digit Med 2018;1:12 [FREE Full text] [CrossRef] [Medline]
- Armstrong KA, Coyte PC, Brown M, Beber B, Semple JL. Effect of Home Monitoring via Mobile App on the Number of In-Person Visits Following Ambulatory Surgery: A Randomized Clinical Trial. JAMA Surg 2017 Jul 01;152(7):622-627 [FREE Full text] [CrossRef] [Medline]
- Dahlberg K, Philipsson A, Hagberg L, Jaensson M, Hälleberg-Nyman M, Nilsson U. Cost-effectiveness of a systematic e-assessed follow-up of postoperative recovery after day surgery: a multicentre randomized trial. Br J Anaesth 2017 Nov 01;119(5):1039-1046 [FREE Full text] [CrossRef] [Medline]
- Belarmino A, Walsh R, Alshak M, Patel N, Wu R, Hu JC. Feasibility of a Mobile Health Application To Monitor Recovery and Patient-reported Outcomes after Robot-assisted Radical Prostatectomy. Eur Urol Oncol 2019 Jul;2(4):425-428. [CrossRef] [Medline]
- Hwang JH, Hee Hwang J, Mun GH. An evolution of communication in postoperative free flap monitoring: using a smartphone and mobile messenger application. Plast Reconstr Surg 2012 Jul;130(1):125-129. [CrossRef] [Medline]
- Chang P, Lin L, Zhang H, Zhao Y, Xie J, Yu Y, et al. Effect of smartphone application assisted medical service on follow-up adherence improvement in pediatric cataract patients. Graefes Arch Clin Exp Ophthalmol 2018 Oct;256(10):1923-1931. [CrossRef] [Medline]
- deBronkart D. From patient centred to people powered: autonomy on the rise. BMJ 2015 Feb 10;350:h148. [CrossRef] [Medline]
- Constantinos CK, Kim DJ. A Meta-Analytical Review of Empirical Mobile Usability Studies. J Usability Stud 2011;6(3):117-171 [FREE Full text] [CrossRef]
- Zapata BC, Fernández-Alemán JL, Idri A, Toval A. Empirical studies on usability of mHealth apps: a systematic literature review. J Med Syst 2015 Feb;39(2):1. [CrossRef] [Medline]
- Schnall R, Mosley JP, Iribarren SJ, Bakken S, Carballo-Diéguez A, Brown IW. Comparison of a User-Centered Design, Self-Management App to Existing mHealth Apps for Persons Living With HIV. JMIR Mhealth Uhealth 2015;3(3):e91 [FREE Full text] [CrossRef] [Medline]
- McCurdie T, Taneva S, Casselman M, Yeung M, McDaniel C, Ho W, et al. mHealth consumer apps: the case for user-centered design. Biomed Instrum Technol 2012 Sep;Suppl:49-56 [FREE Full text] [CrossRef] [Medline]
- Krebs P, Duncan DT. Health App Use Among US Mobile Phone Owners: A National Survey. JMIR Mhealth Uhealth 2015 Nov;3(4):e101 [FREE Full text] [CrossRef] [Medline]
- Boulos MNK, Brewer AC, Karimkhani C, Buller DB, Dellavalle RP. Mobile medical and health apps: state of the art, concerns, regulatory control and certification. Online J Public Health Inform 2014;5(3):229 [FREE Full text] [CrossRef] [Medline]
- Nassar V. Common criteria for usability review. Work 2012;41 Suppl 1:1053-1057. [CrossRef] [Medline]
- Baharuddin R, Singh D, Razali R. Usability Dimensions for Mobile Applications-A Review. RJASET 2013 Feb 21;11(9):2225-2231. [CrossRef]
- ISO 9241. International Organisation for Standardization. URL: https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en [accessed 2019-11-08]
- Harrison R, Flood D, Duce D. Usability of mobile applications: literature review and rationale for a new usability model. J Interact Sci 2013;1(1):1. [CrossRef]
- Lewis JR. IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction 1995 Jan;7(1):57-78. [CrossRef]
- Brooke J. SUS - A quick and dirty usability scale. Usability Evaluation In Industry 1996 Jun 11:189-194 [FREE Full text]
- Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015 Mar;3(1):e27 [FREE Full text] [CrossRef] [Medline]
- Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and Validation of the User Version of the Mobile Application Rating Scale (uMARS). JMIR Mhealth Uhealth 2016 Jun 10;4(2):e72 [FREE Full text] [CrossRef] [Medline]
- Zhao B, Tai-Seale M, Longhurst C, Clay B. Utilization of Hospital Room Hospitality Features on Patient-Controlled Tablet Computers: Cohort Study. JMIR Mhealth Uhealth 2019 Jun 20;7(6):e13964 [FREE Full text] [CrossRef] [Medline]
- Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017 Sep 21;358:j4008 [FREE Full text] [CrossRef] [Medline]
- Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009 Jul 21;339:b2535 [FREE Full text] [CrossRef] [Medline]
- Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan-a web and mobile app for systematic reviews. Syst Rev 2016 Dec 05;5(1):210 [FREE Full text] [CrossRef] [Medline]
- Timmers T, Janssen L, van der Weegen W, Das D, Marijnissen W, Hannink G, et al. The Effect of an App for Day-to-Day Postoperative Care Education on Patients With Total Knee Replacement: Randomized Controlled Trial. JMIR Mhealth Uhealth 2019 Oct 21;7(10):e15323 [FREE Full text] [CrossRef] [Medline]
- Yadav SK, Jha CK, Mishra SK, Mishra A. Smartphone-Based Application for Tele-follow-up of Patients with Endocrine Disorders in Context of a LMIC: A Compliance, Satisfaction, Clinical Safety and Outcome Assessment. World J Surg 2020 Feb;44(2):612-616. [CrossRef] [Medline]
- Ramkumar PN, Haeberle HS, Ramanathan D, Cantrell WA, Navarro SM, Mont MA, et al. Remote Patient Monitoring Using Mobile Health for Total Knee Arthroplasty: Validation of a Wearable and Machine Learning-Based Surveillance Platform. J Arthroplasty 2019 Oct;34(10):2253-2259. [CrossRef] [Medline]
- Argent R, Slevin P, Bevilacqua A, Neligan M, Daly A, Caulfield B. Wearable Sensor-Based Exercise Biofeedback for Orthopaedic Rehabilitation: A Mixed Methods User Evaluation of a Prototype System. Sensors (Basel) 2019 Jan 21;19(2) [FREE Full text] [CrossRef] [Medline]
- Brunner TH, DiFortuna K, LeTang M, Murphy J, Stemplewicz K, Kovacs M, et al. Feasibility of an iPad to Facilitate Communication in Postoperative Patients With Head and Neck Cancer. J Perianesth Nurs 2018 Aug;33(4):399-406 [FREE Full text] [CrossRef] [Medline]
- van der Meij E, Huirne JA, Ten Cate AD, Stockmann HB, Scholten PC, Davids PH, et al. A Perioperative eHealth Program to Enhance Postoperative Recovery After Abdominal Surgery: Process Evaluation of a Randomized Controlled Trial. J Med Internet Res 2018 Jan 02;20(1):e1 [FREE Full text] [CrossRef] [Medline]
- Felbaum DR, Stewart J, Anaizi A, Sandhu FA, Nair MN, Voyadzis JM. Implementation and Evaluation of a Smartphone Application for the Perioperative Care of Neurosurgery Patients at an Academic Medical Center: Implications for Patient Satisfaction, Surgery Cancelations, and Readmissions. Oper Neurosurg (Hagerstown) 2018 Mar 01;14(3):303-311. [CrossRef] [Medline]
- Goz V, Anthony C, Pugely A, Lawrence B, Spina N, Brodke D, et al. Software-Based Postoperative Communication With Patients Undergoing Spine Surgery. Global Spine J 2019 Feb;9(1):14-17 [FREE Full text] [CrossRef] [Medline]
- Gunter RL, Fernandes-Taylor S, Rahman S, Awoyinka L, Bennett KM, Weber SM, et al. Feasibility of an Image-Based Mobile Health Protocol for Postoperative Wound Monitoring. J Am Coll Surg 2018 Mar;226(3):277-286 [FREE Full text] [CrossRef] [Medline]
- Gustavell T, Langius-Eklöf A, Wengström Y, Segersvärd R, Sundberg K. Development and Feasibility of an Interactive Smartphone App for Early Assessment and Management of Symptoms Following Pancreaticoduodenectomy. Cancer Nurs 2019;42(3):E1-E10. [CrossRef] [Medline]
- Harder H, Holroyd P, Burkinshaw L, Watten P, Zammit C, Harris PR, et al. A user-centred approach to developing bWell, a mobile app for arm and shoulder exercises after breast cancer treatment. J Cancer Surviv 2017 Dec;11(6):732-742 [FREE Full text] [CrossRef] [Medline]
- Higgins J, Semple J, Murnaghan L, Sharpe S, Theodoropoulos J. Mobile Web-Based Follow-up for Postoperative ACL Reconstruction: A Single-Center Experience. Orthop J Sports Med 2017 Dec;5(12):2325967117745278 [FREE Full text] [CrossRef] [Medline]
- Highland KB, Tran J, Edwards H, Bedocs P, Suen J, Buckenmaier CC. Feasibility of App-Based Postsurgical Assessment of Pain, Pain Impact, and Regional Anesthesia Effects: A Pilot Randomized Controlled Trial. Pain Med 2019 Aug 01;20(8):1592-1599. [CrossRef] [Medline]
- Khanwalkar AR, Shen J, Kern RC, Welch KC, Smith SS, Tan BK, et al. Utilization of a novel interactive mobile health platform to evaluate functional outcomes and pain following septoplasty and functional endoscopic sinus surgery. Int Forum Allergy Rhinol 2019 Apr;9(4):345-351. [CrossRef] [Medline]
- Mata J, Pecorelli N, Kaneva P, Moldoveanu D, Gosselin-Tardiff A, Alhashemi M, et al. A mobile device application (app) to improve adherence to an enhanced recovery program for colorectal surgery: a randomized controlled trial. Surg Endosc 2020 Feb;34(2):742-751. [CrossRef] [Medline]
- Nilsson U, Dahlberg K, Jaensson M. The Swedish Web Version of the Quality of Recovery Scale Adapted for Use in a Mobile App: Prospective Psychometric Evaluation Study. JMIR Mhealth Uhealth 2017 Dec 03;5(12):e188 [FREE Full text] [CrossRef] [Medline]
- Pecorelli N, Fiore JF, Kaneva P, Somasundram A, Charlebois P, Liberman AS, et al. An app for patient education and self-audit within an enhanced recovery program for bowel surgery: a pilot study assessing validity and usability. Surg Endosc 2018 May;32(5):2263-2273. [CrossRef] [Medline]
- Sousa CS, Turrini R. Development of an educational mobile application for patients submitted to orthognathic surgery. Rev Lat Am Enfermagem 2019 Jul 18;27:e3143 [FREE Full text] [CrossRef] [Medline]
- Sun T, Dunsmuir D, Miao I, Devoy GM, West NC, Görges M, et al. In-hospital usability and feasibility evaluation of Panda, an app for the management of pain in children at home. Paediatr Anaesth 2018 Oct;28(10):897-905. [CrossRef] [Medline]
- Tsapepas DS, Salerno D, Jandovitz N, Hammad S, Jordan P, Mohan S, et al. Using technology to enhance medication regimen education after solid organ transplantation. Am J Health Syst Pharm 2018 Dec 01;75(23):1930-1937. [CrossRef] [Medline]
- Scott AR, Alore EA, Naik AD, Berger DH, Suliburk JW. Mixed-Methods Analysis of Factors Impacting Use of a Postoperative mHealth App. JMIR Mhealth Uhealth 2017 Feb 08;5(2):e11 [FREE Full text] [CrossRef] [Medline]
- Warren-Stomberg M, Jacobsson J, Brattwall M, Jildenstål P. At-home monitoring after surgery/anaesthesia - a challenge. J Eval Clin Pract 2016 Dec;22(6):882-886. [CrossRef] [Medline]
- Debono B, Bousquet P, Sabatier P, Plas J, Lescure J, Hamel O. Postoperative monitoring with a mobile application after ambulatory lumbar discectomy: an effective tool for spine surgeons. Eur Spine J 2016 Nov;25(11):3536-3542. [CrossRef] [Medline]
- Gunter R, Fernandes-Taylor S, Mahnke A, Awoyinka L, Schroeder C, Wiseman J, et al. Evaluating Patient Usability of an Image-Based Mobile Health Platform for Postoperative Wound Monitoring. JMIR Mhealth Uhealth 2016 Sep 28;4(3):e113 [FREE Full text] [CrossRef] [Medline]
- Ponce BA, Brabston E, Zu S. Telemedicine with mobile devices and augmented reality for early postoperative care. 2016 Presented at: 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); Aug 16-20, 2016; Orlando, FL. [CrossRef]
- Jiang Y, Sereika S, Dabbs AD, Handler S, Schlenk E. Acceptance and Use of Mobile Technology for Health Self-Monitoring in Lung Transplant Recipients during the First Year Post-Transplantation. Appl Clin Inform 2016;7(2):430-445 [FREE Full text] [CrossRef] [Medline]
- Chai YJ, Song J, Kang J, Woo J, Song R, Kwon H, et al. A comparative study of postoperative pain for open thyroidectomy versus bilateral axillo-breast approach robotic thyroidectomy using a self-reporting application for iPad. Ann Surg Treat Res 2016 May;90(5):239-245 [FREE Full text] [CrossRef] [Medline]
- Shellmer DA, Dew MA, Mazariegos G, DeVito Dabbs A. Development and field testing of Teen Pocket PATH(®), a mobile health application to improve medication adherence in adolescent solid organ recipients. Pediatr Transplant 2016 Feb;20(1):130-140 [FREE Full text] [CrossRef] [Medline]
- Sun T, West N, Ansermino JM, Montgomery CJ, Myers D, Dunsmuir D, et al. A smartphone version of the Faces Pain Scale-Revised and the Color Analog Scale for postoperative pain assessment in children. Paediatr Anaesth 2015 Dec;25(12):1264-1273. [CrossRef] [Medline]
- Jaensson M, Dahlberg K, Eriksson M, Grönlund Å, Nilsson U. The Development of the Recovery Assessments by Phone Points (RAPP): A Mobile Phone App for Postoperative Recovery Monitoring and Assessment. JMIR Mhealth Uhealth 2015 Sep 11;3(3):e86 [FREE Full text] [CrossRef] [Medline]
- Symer MM, Abelson JS, Milsom J, McClure B, Yeo HL. A Mobile Health Application to Track Patients After Gastrointestinal Surgery: Results from a Pilot Study. J Gastrointest Surg 2017 Sep;21(9):1500-1505. [CrossRef] [Medline]
- Semple JL, Sharpe S, Murnaghan ML, Theodoropoulos J, Metcalfe KA. Using a mobile app for monitoring post-operative quality of recovery of patients at home: a feasibility study. JMIR Mhealth Uhealth 2015 Feb 12;3(1):e18 [FREE Full text] [CrossRef] [Medline]
- Bini S, Mahajan J. Clinical outcomes of remote asynchronous telerehabilitation are equivalent to traditional therapy following total knee arthroplasty: A randomized control study. J Telemed Telecare 2017 Feb;23(2):239-247. [CrossRef] [Medline]
- Smartwatch Market - Growth, Trends, Forecasts. Mordor Intelligence. URL: https://www.mordorintelligence.com/industry-reports/smartwatch-market [accessed 2020-01-28]
- Torous J, Firth J. The digital placebo effect: mobile mental health meets clinical psychiatry. Lancet Psychiatry 2016 Feb;3(2):100-102. [CrossRef] [Medline]
- Pham Q, Wiljer D, Cafazzo JA. Beyond the Randomized Controlled Trial: A Review of Alternatives in mHealth Clinical Trial Methods. JMIR Mhealth Uhealth 2016 Sep 09;4(3):e107 [FREE Full text] [CrossRef] [Medline]
- Zhou L, Bao J, Setiawan IMA, Saptono A, Parmanto B. The mHealth App Usability Questionnaire (MAUQ): Development and Validation Study. JMIR Mhealth Uhealth 2019 Apr 11;7(4):e11500 [FREE Full text] [CrossRef] [Medline]
|CSUQ: Computer System Usability Questionnaire|
|ISO: Organization for Standardization|
|MAUQ: mHealth App Usability Questionnaire|
|mHealth: mobile health|
|PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses|
|SUS: System Usability Scale|
|uMARS: user version of the Mobile App Rating Scale|
Edited by R Lee; submitted 03.04.20; peer-reviewed by M Tai-Seale, J Hitt, Y Chu; comments to author 07.05.20; revised version received 21.05.20; accepted 26.05.20; published 20.07.20Copyright
©Ben Patel, Arron Thind. Originally published in JMIR Perioperative Medicine (http://periop.jmir.org), 20.07.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Perioperative Medicine, is properly cited. The complete bibliographic information, a link to the original publication on http://periop.jmir.org, as well as this copyright and license information must be included.