The preferred modality (electronic vs paper) was reported in 14 studies [23,25,31,32,35-38,40,42,45,50], and electronic administration was preferred in 11 studies [23,25,35-38,40,42,45-47]. One study reported a significantly greater preference for the tablet-delivered questionnaires (z=4.96, SE 3.428, P<.001) [49]. Another study asked patients to rate which mode of administration was the most pleasant and least difficult to use with a Likert scale [44]. Overall, of the 16 studies that reported user preference [23,24,31,32,35-38,40,42,44-47,49,50], a preference for ePROM was reported in 13 studies [23,24,35-38,40,42,44,49]. An overview of the reported percentages can be found in Figure 2.

Additionally, 4 [23,32,42,49] of the 16 studies reported sociodemographic variables that significantly influenced the preference for electronic administration (Table 3).

The satisfaction with and attitude towards ePROMs were reported in 7 studies. Most patients who were exposed to an ePROM found it easy to learn, easy to use, would recommend it to other patients, and would like to continue using it [20,21,33,34,43,47]. In a feasibility and acceptability study of a smartphone app for seizure self-management, patients with epilepsy thought ePROMs would reduce medical visits and health-related costs. Positive satisfaction levels with ePROMs were found for people who were younger (P=.002), lived in a city (P<.001), had higher education levels (P=.001), had stable employment (P<.001), had more frequent seizures (P=.01), had poor medication adherence, and owned a smartphone (P=.001) [21]. In breast cancer patients, willingness to use ePROM was higher in the group with previous experience with ePROM than in the group with previous experience with only paper PROM (92.3% and 59%, respectively, P=.001) [26]. Finally, reviewing the results with a health care professional was associated with 6.6-fold increased odds (P<.001) of perceiving systematic ePROMs as a benefit [33].

Time to complete electronic and paper-based questionnaires was reported in 9 studies [35-37,40,43,45-48], and 3 of these studies reported no significant differences in completion time [35,47]. In one study, however, subjects reported that the completion time for the electronic variant was more acceptable (P=.02) and was perceived as less of a barrier (P=.003) compared to the paper version [36]. Significantly lower times for the electronic variant were reported in 3 other studies [37,40,43]. Only 2 of the 9 studies reported significantly lower completion times for the paper version [45,46], owing to the longer log-on procedure required for the ePROM [45]. One study was indecisive. A detailed overview of the completion times can be found in Table 4. Overall, the completion times for ePROMs were at least equal to or faster than those for paper forms.

Engan et al [23] calculated and compared the human resource (HR) costs, specifically the time spent by an employee preparing, receiving, and handling data, of web-based and paper-based questionnaires. The mean HR cost for the web version was 9.5 minutes, whereas the mean HR cost for the paper version was 24 minutes.

Based on a cohort of 500,000 subjects [36], the financial costs of a paper-based questionnaire were calculated, including printing, mailing, returns, and double data entry. In total, it cost €4,965,833 (€9.94/subject) to use a paper-based version. In comparison, the development of a web-based tool by professionals was estimated to cost only €150,000 (€0.3/subject) or just 3% of the amount of the paper version.

Overall, these results indicate that digital data collection is less expensive, especially with large sample sizes, and it reduces HR-related costs.

Of the 10 studies [23,31,35,36,38,39,41,42,45,50] that reported on missing and incomplete data, 7 studies [23,31,36,38,39,41,50] indicated that electronic methods are associated with less missing data and more complete data. Integrated controls embedded in their ePROM administration was reported by 3 articles [23,35,36]. When a question wasn’t answered, an alert message provided the option to revise the answer prior to submission. As such, data entry mistakes in the form of missing, inconsistent, or abnormal values could theoretically be reduced to zero [23,36]. Regarding unanswered questions or incomplete questionnaires, 2 studies reported no statistically significant differences between ePROMs and paper PROMs [35,45]. One study [42] found significantly more missing items in the electronic version. And, one study reported that the answers were more detailed in 4 of 5 open questions on their electronic questionnaire (P<.05) [31]. Details of these results can be found in Table 5. Based on these results, we conclude that data quality is higher with ePROMs.

A retrospective cohort analyzed the annual data from PROM non-completers. PROM monitoring was completed via paper until 2010, and in 2011, ePROMs were implemented. The initial rate of PROM non-completers was 43%-58%. This decreased to less than 20% since the implementation of ePROMs in 2011 [28]. One randomized controlled trial reported response rates of 17.9% in the internet group and 73.2% in the paper group. After sending a reminder, response rates were 64.2% and 76.5%, respectively (risk difference 12.2%, P=.002) [50]. Another study found no differences in completion rates between ePROMs and paper PROMs (P=.208) [35].

There is conflicting evidence on the effect of electronic data collection on response rates and adherence. Adherence to ePROM declines over time [20,29]. The opportunity to send automated reminders (eg, email or notification) to subjects can improve response rates and compliance [20,50].

The role of ePROMs in symptom management and decision making was acknowledged in multiple studies. Andikyan et al [20] and Schnall et al [22] reported that electronic symptom self-reporting was important in clinical decision making. Automated data collection and processing via ePROM can generate automated alerts to health care professionals when a patient reports disturbing or severe symptoms [20]. It allows early detection of complications, immediate action, and potentially reduction in symptom burden, complications, and readmissions to the hospital. Furthermore, it empowers patients and improves patient-clinician communication [22,24,42]. This is facilitated by the opportunity to plot results visually with a graph or visual aids and gives both the patient and clinicians better insight in the evolution of the patient’s health status [25,34,43].

ePROMs have the advantage of always being available [24,25]. There is no paper waste [34,41], and ePROMs are portable and can be used to measure across multiple devices [42,46,49]. These reported ‘other benefits’ originate from studies with the lowest methodological quality.

As of May 25, 2018, all European organizations are expected to be compliant with the General Data Protection Regulations. This is reassurance for patients that the law is on their side when it comes to the use of their personal health data. All included articles and studies were performed before the implementation of the General Data Protection Regulations. However, privacy concerns were reported in 2 studies [21,26]. Liu et al [21] reported that the majority of patients (71.7%) thought their privacy should be adequately protected. In another study, patients were asked whether there were any barriers related to privacy and technology that would negatively influence their willingness to use ePROMs, and 30% were concerned about privacy issues. The study showed that barriers can be overcome by exposing the patients to an ePROM, which significantly influenced the willingness to participate in electronic assessments [26].

Disadvantages due to technical issues were addressed in 5 articles. The difficulty of or problems with login procedures were addressed in 3 studies [24,45]. Furthermore, technical difficulties adversely impacted compliance; patients who experienced technical difficulties completed fewer daily symptom entries (41.0%) than those who did not (76.0%) [29]. In another study, the needs and possible technological support structures were investigated. The importance of different possible support services to help complete a web-based questionnaire was assessed. Onsite support services were rated as being moderately or highly important by 38%. Technical telephone support was rated as moderately important or very important by 52%. At least 61% would appreciate receiving direct feedback after using the ePROM app [26].

Electronic data collection may require a large initial financial investment (eg, to purchase tablets or computer infrastructure and software, equipment costs, hiring computer programmers, or accessing cellular internet) [30,36,45].

A major disadvantage of ePROM is the potential of a ‘digital divide’. People who are computer illiterate, are older, or have no access to infrastructure could be disadvantaged. One study reported that more than 50% of >70 year olds were not able to complete the electronic version without assistance due to computer illiteracy; less assistance was required for patients completing the paper version [27]. In a second study, patients who needed support were significantly older [42]. The digital divide was also illustrated in another study with cancer patients. Patients who refused ePROM or chose phone calls over (home-based) ePROMs were approximately 10 years older. Patients may differ in terms of available internet, user experience, and affinity for new media. Older or computer-illiterate patients need opportunities to familiarize themselves with the devices [24]. Older patients with poorer health-related quality of life and fewer pre-existing technical skills reported barriers for ePROMs more frequently [26]. Wintner et al [24] reported that patients found ePROMs too impersonal.

Suggestions and tips for ePROM apps were extracted from 12 studies [20,21,24,27,29,30,32-35,38,42]. ePROMs should be free, simple, and minimalistic. They should have a good design, good user experience, adjustable font size, and adaptable user interface. When you start implementing ePROMs, provide educational sessions or support, think of the link with electronic health records, and review the results of the ePROMs with the patients because of the increased perception of benefit. ePROMs should provide positive reinforcement for the patients. Based on our results and discussion, we created a comprehensive overview of the benefits, disadvantages, and suggestions for ePROMs (see Figure 3).

The goal of this systematic review was to systematically and critically summarize the evidence on the use of ePROMs and find the potential benefits and disadvantages. We conclude that ePROM collection is feasible and accepted in healthy people and a wide range of patients with different conditions. Taking into account the results from the strongest methodological studies and the items that were reported in multiple studies, electronic data collection is preferred over paper-based collection, costs less, improves data quality, results in similar or faster completion times, and requires less administration time. Clinical decision making in combination with adequate symptom management can be facilitated. Expressed opinions reflected positive thoughts and attitudes towards ePROMs. Overall, participants found it easy to use, found it easy to learn, and would recommend it to others.

Although our findings are generally favorable towards ePROMs, we cannot ignore the potential disadvantages. Aspects to consider are privacy protection, the one-time large financial investment, and exclusion of certain populations. Patients may be unwilling or unable to complete ePROMs due to higher age or computer illiteracy. Some patients have no internet access, do not have technological devices, or are not acquainted with technological devices. These reported disadvantages and barriers need to be considered when implementing a digital data collection tool in any population. Potential solutions may include an educational session on the use of the digital app and providing sufficient support [24,27,42]. It is also useful to at least provide back-up pen-and-paper data collection to avoid excluding segments of the population from receiving the best possible health care [20,32]. Several suggestions to keep in mind when creating an ePROM are also mentioned in this literature review, which could increase patient experience, usability, and acceptability.

Considering the influence of age, 2 studies suggest that it is an important factor that could potentially increase completion time [40,46]. In contrast, one study found no relationships between completion time and computer skills, age, or education [37]. Older people in particular have reservations concerning modern computer technology and need to be properly approached, especially since we found that younger people had a significantly greater preference for ePROMs [23,32,42]. In our systematic review, we found that various groups of patients with a chronic disease preferred ePROMs over paper versions. On the aspect of completion time, only the time for the patient to complete the questionnaire was measured in the included articles. However, one of the greatest reported advantages of electronic data collection is automated data processing [36,38,41-43,45,49], which subsequently reduces HR time [23], and data are less prone to administration errors. Clinical-based decision-making models using daily registration of PROMs can thus be created.

The strengths of this literature review are that 32 studies concerning the research question were retrieved. Not all studies were comparative trials but assessed patient satisfaction or attitude towards a single ePROM [21,24,25,32,33]. These studies, although not methodologically the strongest, provided capital insights for the research question.

In this systematic literature search, we only searched two databases. It is, therefore, possible that we missed some clinical studies. Moreover, the limited methodological quality of some of the included studies diminished the power of the recommendations.

The overall methodological quality of the included articles was moderate. Disadvantages were a lack of blinding of participants, heterogeneity of outcome measures, heterogeneity of patient populations, different ePROM questionnaires, and different ePROM modalities/formats. Generalizing or comparing results is therefore more difficult, and the results should be interpreted with caution.

The most frequently used screen-based device was tablets. This may be because tablet screens are larger than traditional handheld devices, are easy to use, and can be used for device-based systems. They can provide access to web-based portals or can be used with downloadable apps, which makes them the primary platform for site-based (ie, hospital, care centers) ePROM collection. On the contrary, desktops usually lack touch screen functionality and require the use of a keyboard and/or mouse to respond to questions [14]. Different electronic modes were used in the different articles. The advantages and disadvantages of the different electronic modes are difficult to conclude from this study. Contrasting evidence was found in previously published literature. Two reviews reported their concerns of equivalence between different electronic modes [8,10]; however, White et al [10] found small differences in the correlations, which were not significant regardless of the electronic mode used. In clinical trials, multiple modes of administration may be used, and new findings may be compared to findings that used a different electronic mode of data collection. Further research is warranted regarding the influence of the electronic mode on measurement equivalence. Our findings predominantly complement those from other published literature. Belisario et al [53] conducted a review to assess the impact of apps on the quality of survey questionnaire responses and reported contradictory results regarding completion times but acknowledged that apps might improve data completeness with more complete records than paper administration. Similar to our findings, they reported that there is not enough evidence that apps impact adherence to sampling protocols. Muehlhausen et al [9] conducted a meta-analysis on the equivalence of electronic and paper administration of PROMs and showed that ePROMs yielded comparable results to those of the paper-based variant. Their findings also confirmed the ISPOR taskforce’s conclusion that full psychometric testing of new ePROMs is not necessary for migrations with minor changes only [12]. For researchers and sponsors, this is a clinically and financially reassuring aspect that might facilitate the decision-making process to migrate from paper to digital data collection. The bring-your-own-device (BYOD) approach for ePROM data collection shows potential. BYOD allows participants to use their own computer device (eg, smartphone, tablet, laptop) to access and complete ePROMs [14]. However, there are still a number of issues (eg, software, security, ownership) that need to be resolved before BYOD becomes widely used.

The importance of PROMs is widely accepted. Collecting PROMs with paper-based questionnaires requires many subsequent time-consuming steps [45] that hamper wide implementation in daily care. Electronic collection of PROMs overcomes many of these steps. The potential to collect, score, analyze, visualize, and almost instantly review the results may facilitate workflow. Clinically, we believe ePROMs will improve the interchangeability of information between health care workers, patient-clinician communication, and patient care due to its always available nature. In addition, automated data processing in combination with targeted strategies (eg, automated alerts when patients report disturbing symptoms) has major clinical implications. Clinicians and researchers will also benefit from digital data collection since it reduces administration time. Furthermore, integration of ePROMs into electronic health records may be fundamental to advancing clinical care to improve patient engagement and health outcomes.

Based on this study, we found multiple advantages for the use of ePROMS in several fields of care. ePROMs are preferred over paper-based forms, cost less, improve data quality, result in similar or faster completion times, reduce administration times, and facilitate clinical decision making in combination with adequate symptom management. Subjects expressed positive thoughts and attitudes towards electronic data collection. Potential disadvantages have been mapped but they are not of the magnitude to disregard ePROMs. Furthermore, suggestions have been provided to counteract the disadvantages. This review allows researchers and clinicians to consider both the advantages and disadvantages of selecting one mode over the other. While electronic modes offer advantages for all involved parties (eg, patients, hospitals, government), implementing (new) ePROMs requires careful considerations of the implications on the study population and may require additional steps (eg, provision of internet access, acquiring electronic devices) to include participants who would be excluded otherwise.