The consensus level was determined to be 70%, similar to other related studies in health research [19-21]. Descriptive statistics were used to analyze the experts’ opinions, using frequencies of responses for questions that were not open-ended. If the frequency was 70% or higher, the experts were deemed to have reached consensus on this particular response.

All responses were considered in the analysis; however, consensus was calculated based on how many experts answered each question. Partially filled responses were also included, as other published studies have done in the past [22,23].

The strategy for an event of nonconsensus was to critically evaluate and discuss the respondents’ answers and to revise the questions in the subsequent rounds.

Round 1 of the study largely included open-ended questions to determine a variety of ways the experts would choose to define and categorize complications following cardiac surgery. The thematic analysis framework [24] was used to analyze the responses to the open-ended questions, and the results were included as options for responses in the subsequent round of the study as in the Delphi method [25].

A sample of the data was coded separately by 3 researchers to ensure coding coherence and consistency. Once coding consistency was established through the initial sample coding, data coding was conducted by 1 researcher (LL). Coding consistency and thematic analysis were subsequently discussed, and conflicts were resolved at regular meetings of the study investigative team, which includes substantial expertise in mixed methods and qualitative research (MMB).

Following the guidance of Hasson et al [25], statements that were identified as identical or similar were grouped as common concepts. Once specific themes were created, the statements within a thematic group were synthesized into a single summary statement after discussion between the study investigators. The wording was kept as close as possible to the statements that had been provided by the experts. Any unique statement provided by the experts with no related statement was kept as worded originally and included directly in Round 2.

Comments were received from 50 experts on how they would define the term “Postoperative Complication” in cardiac surgery. The definitions emerging from Round 1 of the study were then used in the Round 2 questionnaire to reach consensus on a single definition.

All proposed definitions focused on different impacts of complications on the patient, institution, and surgery itself (eg, delayed recovery, impact on patient’s quality of life, and hospital length of stay). Hence, for simpler analysis, these statements were analyzed thematically [24] and categorized under themes based on the definitions that the experts offered. For example, the concept of “An unplanned adverse event occurring after cardiac surgery that may be caused or compounded by the surgical process” included statements such as “The event can be unplanned,” “The event must be harmful or unfavorable,” “The complication must be present following cardiac surgery, specifically,” and “The event must occur after surgery and is unlikely to occur if the patient did not have the surgery.” These common themes were then grouped and synthesized under common characteristics of the complications such as “unplanned,” “adverse event,” “cardiac surgery,” and “surgery.” All characteristics and themes can be found in Table 1.

The responses for each definition were then mapped onto each characteristic to find what the experts deemed important to define what constitutes a postoperative complication following cardiac surgery, which could then be used for conducting Round 2 of the Delphi study.

Responses to the question as to whether they thought it is useful to define and classify postoperative complications for cardiac surgery were provided by 51 experts. Of these 51 experts (Table 2), 23 (45%) thought it was “Extremely useful,” and 20 (39%) thought it is “Very useful.” Combining these percentages, based on the predetermined consensus level of 70%, it can be concluded that the experts have reached the consensus that it is very useful to classify postoperative complications for cardiac surgery, with a consensus level of 84%.

The experts provided various reasons why they thought it is useful to classify postoperative complications for cardiac surgery, which included improving audit and quality measurement, helping with planning and management, risk management and communications, and helping to improve research in the field. Some of the participants responded in the following ways when asked to explain why defining and classifying complications is useful:

Overall, 48 experts stated how many categories postoperative complications should have. Most of the respondents wanted 3 to 5 grades to categorize complications: Of the 48 respondents, 16 (33%) voted for 3 grades, 12 (25%) voted for 4 grades, and 14 (29%) voted for 5 grades. Some (26/48, 54%) also named the categories they offered, and it became clear that respondents offered the following variations as a common answer:

All 26 experts who provided categories included mild/moderate/severe as the category combination. This means that consensus was reached that the categories for postoperative complications for cardiac surgery will be classified as “Mild,” “Moderate,” and “Severe.” Since many respondents offered “Death” as a separate class, the experts were asked to decide whether to add that to the categories in Round 2 of the study. Since no complication would be categorized as “None,” this was not added to the categories.

Experts also provided possible definitions for each category that they proposed. To analyze the suggested definitions, the thematic analysis, explained in detail in the Qualitative Analysis section, focused on characteristics that each complication category could have. Like in the Defining the Term “Postoperative Complication” section, the characteristics provided by experts for each category of complications were collated so that similar characteristics were merged into one, and unique characteristics were left in their initial form [14]. The final list of characteristics proposed by the experts was as follows: effect on overall length of stay in hospital, effect on final outcome, length of the complication, clinical relevance, impact on the patient, occurrence of the complication, therapeutic intervention required, and impact on the institution

These factors were then related to a level of complication. For example, the question “What is the effect on overall length of stay in hospital?” was converted into “No consequential effect on overall length of stay” for the Mild level of complication, “Some effect on overall length of stay” for the Moderate level of complication, and “Extended length of stay” for the Severe level of complication. These statements were then used in Round 2 of the Delphi study so experts could vote on which characteristics were most important to define each complication category.

The Round 2 survey (see Multimedia Appendix 2) of the Delphi study was sent to the same societies and contact list from the Scottish cardiac centers as described in the Identification of Experts section. To take part in Round 2, the experts were not required to have taken part in Round 1 of the study, as per the “all rounds” approach [15]. Just like in Round 1, the experts had to answer the filter question to make sure they were eligible to participate.

The aims of Round 2 of the study were to reach consensus regarding the following:

The choices for answers for the questions were collated based on the results of Round 1 of the study. Just like in Round 1, descriptive statistics were used to analyze the opinions of experts, using frequencies of responses for questions that were not open-ended. If the frequency of a response was 70% or higher, the experts were deemed to have reached consensus on this particular response.

Round 2 of the questionnaires were sent on June 2, 2020, and a reminder was sent on June 16, 2020. The survey was open for 4 weeks (closed on June 30, 2020).

Overall, 46 experts took part in the survey, and 37 of them finished the survey. As done in the previous round, we also included responses from participants who partially completed the survey in this round.

Experts voted for each characteristic (see the Defining the Term “Postoperative Complication” section) to define what constitutes a complication after cardiac surgery. Consensus was reached that all characteristics (Table 1), apart from “Unexpected,” should be included in the final definition.

Combining these characteristics into a sentence resulted in the following definition:

Of 37 experts, 31 (84%) thought that “Death” should be included in the classification of postoperative complications. As a result, consensus was reached that the complications should be categorized in 4 levels: “Mild,” “Moderate,” “Severe,” and “Death.”

Based on the proposed characteristics that were collated from experts’ responses (described in the Defining the Categories of Postoperative Complications section), consensus was reached on definitions for “Mild” complications (Table 3). Hence, a complication following cardiac surgery is classified as “Mild” if the complication has the following characteristics: The complication has no consequential effect on the final patient outcome (28/37, 76%), and the complication has a minimal impact on the patient (27/37, 73%).

Similarly, as shown in Table 4, a complication following cardiac surgery is classified as “Severe” if the complication is potentially life-threatening (34/37, 92%), there is a consequential or long-standing impact on the patient (31/37, 84%), or a notable amount of intervention is required due to this complication (26/37, 70%).

The experts did not reach consensus on the definition for “Moderate” complications due to none of the characteristics receiving 70% or more of the votes (Table 5). However, one could argue that the definition of moderate is known, as it is neither mild nor severe. This is further discussed in the Limitations section.

Finally, to understand the experts’ understanding of which specific complications could fall into the established complication categories, experts were also asked to provide examples for each proposed complication level, examples of which were hemodynamic instability as a “Mild” complication, atrial fibrillation as a “Moderate” complication, and acute renal failure as a “Severe” complication. A list of examples of complications and how they were categorized by experts can be found in Multimedia Appendix 3. However, since there is currently no single nomenclature for surgical complications, unlike for clinical diagnosis (ie, the International Statistical Classification of Diseases-10), the classifications can vary, especially in the “Moderate” group. Hence, the list of complications and their categories presented in Multimedia Appendix 3 should be interpreted with caution.

In Round 1 and Round 2 of the study, 51 and 37 experts completed the study, respectively. According to publications discussing the Delphi method, both rounds of the study had a sufficiently large sample size, as it does not depend on statistical power but rather on group dynamics for coming to consensus among experts. Hence, an expert panel usually consists of 10 to 30 experts [31]. Furthermore, since this was an e-Delphi study, it can be expected that the experts were not influenced by one another, as the respondents did not know what other respondents had said; therefore, the group dynamic came through each individual from analysis of experts’ responses.

As seen from the results of the study, most experts were cardiac anesthetists and intensivists; however, no cardiac surgeons took part in the study. Historically, the decision as to whether a patient will be operated upon is primarily made by the surgeon. Understanding surgeons’ views on defining and classifying complications in cardiac surgery would be useful. Hence, we have involved surgeons in an ongoing study regarding system requirements for a clinical decision support predicting complications. However, 90% of the participants in this study were involved with decision making, which is common with the creation of preassessment clinics, where decisions about patient care are made by multidisciplinary teams [32].

Although this is a pilot study that aimed to develop a classification system for complications in cardiac surgery, in future work, a more international panel of experts is needed to increase the impact of the classification system. Although experts within the European Association of Cardiothoracic Anaesthesiology and Intensive Care were invited, the majority of the professional societies were UK-based societies, which explains the lack of responses from international experts. Since the standards in cardiac surgery are common internationally [32], it is likely that results would be similar; however, the consensus would be more representative and more reliable to be put into practice. In addition, the societies were mostly related to cardiac anesthesia; only one (The Society for Cardiothoracic Surgery) was specific to cardiac surgeons. This explains why no cardiac surgeons took part in the study. However, it can be expected that, if surgeons took part in this study, the results would be similar due to growing interest in investigating postoperative outcomes other than mortality and an interest by both surgeons and anesthetists in improving patient outcomes beyond survival [33]. Hence, in our future study, cardiac centers will be contacted directly to allow for a more international panel, and more efforts will be directed toward recruiting more cardiac surgeons to participate.

No consensus on the definition of “Moderate” complication was reached. Delphi studies do not always reach consensus on all aspects of the study [34]. Categorization decisions are often made based on the extreme categories rather than on the middle category [35]. This has been addressed with, for example, the American Society of Anesthesiologists (ASA) Classification [36], in which there is no “moderate” category. Historically, there have been concerns about the subjectivity of the ASA status [37], and the same problem can occur with our complication classification. To categorize complications appropriately, actions and consequences of each category need to be considered. With a “Mild” complication, some medicines might have to be administered, for example for urinary retention, but in general, no notable action that requires time and resources is needed. With a “Severe” complication, whether it is kidney failure or a stroke, dialysis or thrombectomy, respectively, might be needed. Both interventions are time-consuming and resource-intensive. When it comes to the moderate category, however, it is uncertain whether it is more on the “Mild” or “Severe” side. On one hand, it is generally unclear regarding what action needs to be taken; on the other hand, it provides the users with a spectrum of categories and therefore the possibility to offer more nuance to the problem. As shown by Mayhew et al [37], for the ASA physical status classification, providing example cases for each classification improved objectivity and reduced variability in classification. Hence, we also asked experts to provide examples for each category. However, further work is needed to provide examples; hence, it important to keep in mind that for personalized use, each complication, regardless of which category it falls into, needs an individual treatment approach, depending on the patient’s current state and medical history.