Open Access

Serious electronic games as behavioural change interventions in healthcare-associated infections and infection prevention and control: a scoping review of the literature and future directions

Antimicrobial Resistance & Infection Control20165:34

https://doi.org/10.1186/s13756-016-0137-0

Received: 13 July 2016

Accepted: 28 September 2016

Published: 12 October 2016

Abstract

Background

The uptake of improvement initiatives in infection prevention and control (IPC) has often proven challenging. Innovative interventions such as ‘serious games’ have been proposed in other areas to educate and help clinicians adopt optimal behaviours. There is limited evidence about the application and evaluation of serious games in IPC. The purposes of the study were: a) to synthesise research evidence on the use of serious games in IPC to support healthcare workers’ behaviour change and best practice learning; and b) to identify gaps across the formulation and evaluation of serious games in IPC.

Methods

A scoping study was conducted using the methodological framework developed by Arksey and O’Malley. We interrogated electronic databases (Ovid MEDLINE, Embase Classic + Embase, PsycINFO, Scopus, Cochrane, Google Scholar) in December 2015. Evidence from these studies was assessed against an analytic framework of intervention formulation and evaluation.

Results

Nine hundred sixty five unique papers were initially identified, 23 included for full-text review, and four finally selected. Studies focused on intervention inception and development rather than implementation. Expert involvement in game design was reported in 2/4 studies. Potential game users were not included in needs assessment and game development. Outcome variables such as fidelity or sustainability were scarcely reported.

Conclusions

The growing interest in serious games for health has not been coupled with adequate evaluation of processes, outcomes and contexts involved. Explanations about the mechanisms by which game components may facilitate behaviour change are lacking, further hindering adoption.

Keywords

Serious game Gamification Healthcare-associated infection Implementation Adoption Scoping study

Background

Healthcare-associated infections (HCAIs) affect millions of patients worldwide with significant economic and human costs [1, 2]. To address this challenge, healthcare organisations have implemented multiple improvement strategies with varying success [3]. These include educational programmes [4, 5], performance feedback [6] and guidelines [7], often ‘bundled’ [8].

However, engaging healthcare workers (HCWs) in sustained best practice remains a challenge. For such reason, the exploration of individual [9] and social motivations [10, 11], and the consideration of organisational contexts [12] has been suggested as beneficial for the effective adoption of behaviour change strategies.

The use of innovative interventions such as serious games to encourage optimal clinical behaviours has received increasing attention for their potential to overcome engagement deficits [13]. A ‘serious game’ is defined as an ‘interactive computer application, with or without significant hardware component, that has a challenging goal, is fun to play and engaging, incorporates some scoring mechanism, and supplies the user with skills, knowledge or attitudes useful in reality’ [13]. Serious games have already been used in clinical medicine, surgery and public health with successful results [1416]. Seemingly, game users enjoy interacting with games because they can fulfil psychological needs such as control, autonomy and a sense of achievement [14].

Despite increasing evidence for the role of serious games in several fields, there remains a paucity of data supporting their application within infection prevention and control (IPC). To explore whether serious games could promote HCW behaviour change in this crucial area, we conducted a scoping study aiming to synthesise relevant evidence and identify knowledge gaps in the development and evaluation of game-based interventions.

Methods

Scope

We largely followed the framework suggested by Arksey and O’Malley [17] combining literature searches, iterative study selection and data extraction, qualitative analysis and identification of implications for future research.

Application of analytic framework

The complexity and heterogeneity of serious games may explain the scarcity of evaluative approaches [18]. Although a framework for the evaluation of serious games has been proposed [19], it was constructed following expert opinions and has yet to be validated; in addition, it lacks consideration to the implementation stage for interventions, which have been demonstrated as important for consideration in other models for the development of behaviour interventions [20]. For this reason, we chose to adopt an analytic framework based on the Stage Model for Behavioural Intervention Development [20]. The framework comprises four stages: i) inception, ii) development, iii) small-, and, iv) large-scale implementation, which are iterative and bidirectional (i.e. informing each other) rather than linear and sequential (see Fig. 1).
Fig. 1

Analytic framework for mapping the games along the stages of intervention formulation and evaluation

Although evaluation occurs mainly at the later cycle stages, it is also closely linked to, and draws upon, knowledge and information obtained in preceding stages (see Table 1 for more detail).
Table 1

Evaluation Variables

Process evaluation variables: a) evaluation methods: sampling, duration, data collection (pre-/in-/post-game, use of frameworks, comparison, nature of evaluators (multiplicity, independency), reflexivity on risk of bias (roles of researchers, internal/external validity), etc.; b) identification of active component of game intervention (explanation about mechanisms by which game components may facilitate intended changes); game concept; learning model [18]; behaviour change model [30]; engagement with the game, etc.

Context evaluation variables: a) inner context: characteristics of adopting individual/organisation/setting; structure (e.g. size of organisation); culture (e.g. organisational commitment); cognitive (e.g. psychological safety); characteristics of participants (gamers) such as socio-demographic (e.g. age, gender, ethnicity); expertise/professional background, position, (previous gaming) experience; personality; learning styles; game skills/attitudes; intrinsic/extrinsic motivation, etc.; b) outer context (macro-context): incentives and mandates, networks, environmental stability, socio-political climate, etc [18].

Outcome evaluation variables: a) actual use: users’ perceptions about usability, acceptance, and/or attractiveness of the game intervention; users’ knowledge, attitude, behavioural change; clinical effectiveness, efficacy (e.g. dose – effect relations), cost-benefit, unintended outcomes, etc.; b) application: scalability, sustainability, fidelity [31], adaptation, etc.

Data sources and searches

We interrogated the electronic databases Ovid MEDLINE, Embase Classic + Embase, PsycINFO, Scopus, and the Cochrane Library Database. Additionally, we used Google Scholar to identify additional references. Search terms were developed and refined following a pilot literature scan and through team discussion in November 2015. The following search terms were used and truncated where necessary, with appropriate MeSH terms where possible: ‘serious game’, ‘gamification’, ‘video game’, ‘computer game’, ‘simulation game’, ‘virtual reality’, ‘infection’, ‘hand hygiene’. Inclusion criteria included: 1) papers published up to present, 2) English language, 3) focus on inception, development, implementation, and/or evaluation of serious game interventions in IPC or HCAI prevention and 4) interventions aimed at HCWs (doctors, pharmacists or nurses). Additionally, the exclusion criteria were: 1) interventions aimed at patients or citizens, and 2) non-intervention studies. See Table 2 for search string, with further details in Additional file 1. We ran the search in December 2015.
Table 2

Search string (Ovid MEDLINE, Embase Classic + Embase), 10 Dec 2015a

(((serious gam* or gamification or (video gam* or video game [MeSH]) or computer gam* or simulation gam* or (virtual reality [MeSH] or virtual realit*)) and (infection [MeSH] or infection*)) not HIV).af.

af All Fields

aOn 11 December 2015, we supplemented the search string with ‘hand hygiene’

Study screening and selection

The Covidence [21] platform was used for study screening and selection. Two researchers (ECS/MI) concurrently and independently selected studies (i.e. title and abstract screening, full text review). A third researcher (TMR) resolved disagreements, with final consensus achieved through discussion.

Data extraction, synthesis, and analysis

A data extraction form was populated with relevant information from each source. As this was a scoping study, we chose not to perform quality assessment of the papers. Inter-rater reliability of analysis was ensured by the independent involvement of the main researchers in assessment.

Results

Studies included and data extraction

Our initial search identified 1125 papers. We removed 160 duplicates and excluded 942 papers in the title and abstract screening stage. 23 papers were assessed for full-text eligibility. Subsequently, 19 references were excluded due to irrelevancy (e.g. not focused on games per se, not digital games, not infection focus, not aiming at HCWs, not primary research), and full text unavailability. Our review finally analysed four sources (Additional file 2). The authors of conference abstracts and proceedings were contacted to provide information regarding their experiences, together with their presentations. We received two responses with presentation slides.

Results from the descriptive and analytical exercises are shown in Table 3 and Additional file 3 respectively.
Table 3

Descriptive overview of studies selected

Authors, Year, Ref

Study 1

Sax and Longtin 2011 [22]

Study 2

Vázquez- Vázquez et al. 2011 [23]

Study 3

Castro-Sánchez et al. 2014 [24]

Study 4

Venier et al. 2015 [25]

Type of paper

Conference (presentation)

Conference

Journal article

Conference (presentation)

Origin of the paper

Switzerland (/Canada)

Spain

England

France

Lead (type of organisation)

University hospital

Regional Patient Safety Observatory (Spain)

University

Coordination centre (fighting nosocomial infections)

Paper focusa

i), ii) and iii) – Inception, scoping; design, development; pretesting, refinement; and successful launching described. No evaluation of implementation done besides pretesting.

i) and ii) - Inception, scoping; design, development, implementation (launching), but no pretesting/pilot, evaluation done.

i) and ii) – Inception, scoping; design, development. No pretesting. Future evaluation provided.

i), ii) and iv) – brief description about inception, scoping, development, and more focusing on description about implementation of a large scale survey, and its evaluation. No pretesting/pilot studies.

Name of game

Story-based serious game

Serious for hand hygiene training.

‘On call: antibiotics’

Flu.0

Description of game intervention

Game users can decide where to use hand hygiene and disposable gloves using story-based serious game in which 2 doctors are interacting with different patients during ward rounds. Emotional engagement, role identity development through medical specific distracting plot, and mental simulation. Immediate feedback messages and tracking mechanism of results are also incorporated.

Promotion of hand hygiene using WHO’s ‘Five Moments for Hand Hygiene’ with a ludic approach. A non-risk environment was created without any adverse effects from actions of game users, who have to decide when and how hand hygiene should be performed in a 3D setting with different hotspots. Every decision is followed by feedback to strengthen success or to explain why game users performed incorrectly. Low cognitive erosion to keep the playability.

Serious game for antimicrobial prescribing decisions in virtual hospital patients. Prescribers receive clinical information and have to make diagnostic and therapeutic decisions. They get immediate feedback on performance and wider impacts of prescribing decisions. Personalisation/scores/leader boards and difficulty enhancement mechanisms incorporated in the game to sustain engagement.

Serious game for nurses and doctors to educate 8 key points to know and to do when dealing with one or more patients with flu.

ai) inception, scoping, ideation; ii) design, development, configuration; iii) small-scale implementation (pretesting/piloting), refinement; iv) large-scale, wide implementation, sustainability

Thematic categories

The following themes were identified: a) a nascent field; b) suboptimal user engagement; and c) limited process, context, and outcome evaluations to inform applicability of interventions.
  1. a)

    An initial look at a nascent field

     

All studies were conducted in European countries. No study was carried out before 2011. Our search strategy yielded few published papers, but a high proportion of conference abstracts (3/4).

Most games focused on inception or development and implementation, with pending evaluation [2224]. Only Flu.0 [25] reported on evaluation.

One source discussed timers as engagement mechanism [24]. Other intrinsic motivators such as scores and leader boards [24] or benchmarking [22, 23] were rarely explicit. A ludic approach to WHO’s Five Moments for Hand Hygiene was employed in one study [23]. No intervention incorporated reward mechanisms. Three of the four studies [2224] aimed at learning and behavioural outcomes whereas Flu.0 [25] focused on learning and attitudinal outcomes. One intervention encompassed primary and secondary care [23], with the rest focusing solely on hospitals. Hand hygiene was the most frequently selected topic [22, 23].
  1. b)

    Suboptimal and user engagement in conception and development stages

     

Technical experts were involved in two studies (2/4), concerning hand hygiene [23] and antimicrobial stewardship [24]. However, no study explicitly involved external marketers. Equally, no potential game users were included in preliminary needs assessments. Frontline health professionals co-operated in a usability test with ‘think-aloud’ protocols [22], as well as designing virtual patients in another intervention [24].

We found limited user-centred evaluation or summative evaluation [26] (e.g. comparison between control and intervention groups, or triangulated with other data sources).
  1. c)

    Limited process, context, and outcome evaluations to inform the applicability of game interventions

     

The type of professionals included varied, from doctors [22] to a mixture of nurses and doctors [25] as well as others [23, 24]. Only one study targeting multi-professional groups stratified outcomes by participants’ profession [25]. Effectiveness was reported in two studies, and included changes in users’ perception, knowledge, attitude, and behaviour driven by the intervention. However, no study assessed clinical effectiveness. Intervention fidelity was only dealt with in one study. Implementation variables were either rarely mentioned (in the case of fidelity and unintended outcomes) or absent (as in sustainability). Context evaluation variables such as size of the organisation or participant’s characteristics, as well as process evaluation variables including communication received scant attention.

Regarding other variables such as scalability, Sax and Longtin [22] inferred the applicability of the game to hand hygiene observers, a different target group. Vázquez-Vázquez et al. [23] discussed the potential expansion of the intervention to patients. The Andalusian game developers proposed to export the intervention to other Spanish-speaking countries [23]. However, neither of these was empirically confirmed nor were the interventions culturally adapted.

None of the studies included economic analyses, and just one paper reflected upon the topic at all [24]. Such evaluation would nonetheless be required to assess the viability of mass game deployment.

Discussion

Principal findings

In summary, we identified four published experiences related to game-based behavioural interventions in IPC. The absence of studies before 2011 together with the high number of conference papers (3/4) contributing to our review reinforces the notion of a budding research field.

With the information available, game-based behavioural interventions in IPC did not seem adequately evaluated, and generally lacked appropriate control groups. The experience of the game for the participating individuals appeared to be equally neglected in evaluations. These deficits would particularly affect absolute intervention effects, internal validity and generalisation of findings, and the subtle relationships between the games and implementation settings.

Outcome variables including fidelity, unintended outcomes, or sustainability were scarcely reported. Scalability of game-based interventions needs to be explored further. For example, analysis of outcomes on those beyond the target groups would enhance the internal validity of the intervention. Moreover, no study was found that conducted economic analyses of their intervention. The uncertainty about economic aspects [27] would undoubtedly affect estimates about the sustainability of game-based interventions. The absence of sustainability assessments appears to be particularly concerning, in view of the suggested appeal of serious electronic games as behaviour change interventions, and the rapid yet constant replacement of technologies and software.

Limitations

Our study has various methodological and practical limitations. Relevant studies were small and most papers were presented at conferences, limiting the access to detailed information about the experiences reported. We tried to reduce the impact of such limitation by obtaining information directly from study authors. Our scoping study did not include non-English or grey literature. We focused on scientific publications, inevitably excluding commercial or non-academic papers that may report about games developed within our target area. However, it is unlikely that clinically-focused interventions would be unreported in healthcare conferences or journals. The analytic framework that we used may also have weaknesses, potentially neglecting the informal knowledge transfer that might also play a role in the conception and evaluation of serious games as intervention. However, we opted to use such framework in view of its comparative advantage over the untested tool proposed by Graafland et al [19] and our interest in exploring the implementation features of the experiences identified.

Future directions

Our scoping study indicates that experiences and research into digital games as behaviour change tools in HCAIs and IPC is an emergent field. The experiences reported focused on a narrow set of clinical areas and scenarios, although the limited number of studies included in our study prevents any firm conclusions. The possibilities afforded by games to explore alternative scenarios and immediate consequences of decisions could be exploited much more efficiently by healthcare organisations. The confluence of technologies such as affordable virtual reality consumer goggles [28] or haptic devices [29] with interventions such as serious games can herald a new era for healthcare training.

For example, role-playing scenarios related to antimicrobial decisions could serve to make explicit the influences exerted on antimicrobial prescribing from other clinicians and patients. Additionally, serious games centred on strategic decision-making may increase practitioner preparedness related to containment of outbreaks of highly virulent pathogens. The same platform may offer lessons about risk communication and management. Games could also enhance the training on traditional yet mechanical IPC tasks such as donning of protective suits or decontamination of surfaces. Personnel involved in the response to future international health emergencies could benefit from culturally-sensitive and accurate serious games that would facilitate their rapid immersion in the assignment.

Conclusions

Whilst there is growing interest in using serious games in health as valuable adjunct to conventional education, training and behaviour change interventions Perhaps due to the immaturity of the field effectiveness, process, context, and outcome evaluation are still missing and methodological aspects can generally be much improved. Future experiences should incorporate multi-level perspectives including data from the game per se as well as individual, team, organisation, and system levels, to capture more subtle interactions between the interventions and implementation contexts. This rich mesh of feedback channels may clarify the mechanisms by which game components may facilitate behaviour change. With antimicrobial resistance attracting attention in recent years from international campaigns and national policies and efforts to include antimicrobial stewardship at high-level decision fora, we suspect that this area is likely to flourish globally.

Abbreviations

HCAI: 

Healthcare associated infection

HCWs: 

Healthcare workers

IPC: 

Infection prevention and control

Declarations

Acknowledgements

We would like to thank Nisha Shah (Imperial College London) for her assistance with the initial stages of the study.

Funding

This work was supported by the National Institute for Health Research Health Protection Research Unit (NIHR HPRU) [HPRU-2012–10047] in Healthcare Associated Infection and Antimicrobial Resistance at Imperial College London in partnership with Public Health England. The views expressed are those of the author(s) and not necessarily those of the National Health Service, the NIHR, the Department of Health or Public Health England. The content of this article is solely the responsibility of the authors and any conclusions or interpretations may not reflect the funders’ views.

Availability of data and materials

All data generated or analysed during this study are included in this published article [and its supplementary information files].

Authors’ contributions

ECS devised the study concept and design. YK, ECS, and MI developed the analytic framework. TMR contributed to study selection. ECS and MI analysed data. ECS and MI drafted the manuscript, and YK provided critical revision of the manuscript and contributed to writing the manuscript. ECS, YK, MI, and TMR edited the manuscript. AH provided critical input during all stages of the study. All authors read and approved the final manuscript.

Competing interests

The authors declared that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Not applicable.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
NIHR Health Protection Research Unit in Healthcare Associated Infection & Antimicrobial Resistance at Imperial College London
(2)
Health Services Research & Management Division, School of Health Sciences, City University London

References

  1. World Health Organization. Report on the burden of endemic health care-associated infection worldwide: a systematic review of the literature. Geneva: WHO; 2011. http://apps.who.int/iris/bitstream/10665/80135/1/9789241501507_eng.pdf. Accessed 4 Apr 2016.Google Scholar
  2. Taylor J, Hafner M, Yerushalmi E, et al. Estimating the economic costs of antimicrobial resistance: model and results. Santa Monica, Calif., and Cambridge. UK: RAND Corporation; 2014. Available at: http://www.rand.org/content/dam/rand/pubs/research_reports/RR900/RR911/RAND_RR911.pdf. Accessed 4 Apr 2016.Google Scholar
  3. Davey P, Brown E, Charani E, et al. Interventions to improve antibiotic prescribing practices for hospital inpatients. Cochrane Database of Systematic Reviews 2013; 4. doi: 10.1002/14651858.CD003543.pub3Google Scholar
  4. Butler CC, Simpson SA, Dunstan F, et al. Effectiveness of multifaceted educational programme to reduce antibiotic dispensing in primary care: practice based randomised controlled trial. BMJ. 2012;344:d8173.PubMedPubMed CentralGoogle Scholar
  5. Roque F, Herdeiro MT, Soares S, Teixeira Rodrigues A, Breitenfeld L, Figueiras A. Educational interventions to improve prescription and dispensing of antibiotics: a systematic review. BMC Pub Health. 2014;14:1276.Google Scholar
  6. Gjelstad S, Høye S, Straand J, Brekke M, Dalen I, Lindbæk M. Improving antibiotic prescribing in acute respiratory tract infections: cluster randomised trial from Norwegian general practice (prescription peer academic detailing (Rx-PAD) study). BMJ. 2013;347:f4403.PubMedPubMed CentralGoogle Scholar
  7. Barlam TF, Cosgrove SE, Abbo LM, et al. Implementing an antibiotic stewardship program: guidelines by the Infectious Diseases Society of America and the Society for Healthcare Epidemiology of America. Clin Infect Dis. 2016;Apr 13:Google Scholar
  8. Boyd S, Aggarwal I, Davey P, Logan M, Nathwani D. Peripheral intravenous catheters: the road to quality improvement and safer patient care. J Hosp Infect. 2011;77:37–41.PubMedGoogle Scholar
  9. Carthey J, Walker S, Deelchand V, Vincent C, Griffiths WH. Breaking the rules: understanding non-compliance with policies and guidelines. BMJ. 2011;343:d5283.PubMedGoogle Scholar
  10. Charani E, Castro-Sánchez E, Sevdalis N, et al. Understanding the determinants of antimicrobial prescribing within hospitals: the role of “prescribing etiquette”. Clin Infect Dis. 2013;57:188–96.PubMedPubMed CentralGoogle Scholar
  11. Ferlie E, Fitzgerald L, Wood M, Hawkins C. The nonspread of innovations: the mediating role of professionals. Acad Manage J. 2005;48:117–34.Google Scholar
  12. Grimshaw JM, Thomas RE, MacLennan G, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004;8:iii–iv. 1–72.PubMedGoogle Scholar
  13. Graafland M, Schraagen JM, Schijven MP. Systematic review of serious games for medical education and surgical skills training. Br J Surg. 2012;99:1322–30.PubMedGoogle Scholar
  14. Baranowski T, Buday R, Thompson DI, Baranowski J. Playing for real. video games and stories for health-related behavior change. Am J Prev Med. 2008;34:74–82.PubMedPubMed CentralGoogle Scholar
  15. Plerhoples TA, Zak Y, Hernandez-Boussard T, Lau J. Another use of the mobile device: warm-up for laparoscopic surgery. J Surg Res. 2011;170:185–8.PubMedGoogle Scholar
  16. Badurdeen S, Abdul-Samad O, Story G, Wilson C, Down S, Harris A. Nintendo Wii video-gaming ability predicts laparoscopic skill. Surg Endosc. 2010;24:1824–8.PubMedGoogle Scholar
  17. Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32.Google Scholar
  18. Hainey T, Connolly T, Yaëlle C, Boyle E, Beeby R, Soflano M. Assessment Integration in Serious Games. In T. M. Connolly, T. Hainey, E. Boyle, G. Baxter, & P. Moreno-Ger (Eds.), Psychology, Pedagogy, and Assessment in Serious Games. (pp. 317-341). Hershey: IGI Global; 2014. doi:https://doi.org/10.4018/978-1-4666-4773-2.ch015.
  19. Graafland M, Dankbaar M, Mert A, et al. How to systematically assess serious games applied to health care. JMIR Serious Games. 2014;2:e11. doi:https://doi.org/10.2196/games.3825.PubMedPubMed CentralGoogle Scholar
  20. Onken LS, Carroll KM, Shoham V, Cuthbert BN, Riddle M. Reenvisioning clinical science: unifying the discipline to improve the public health. Clin Psychol Sci. 2014;2:22–34.PubMedPubMed CentralGoogle Scholar
  21. The Covidence. https://www.covidence.org/. Accessed 11 Dec 2015.
  22. Sax HS, Longtin Y. Immersive hand hygiene trainer for physicians – a story-based serious game. BMC Proc. 2011;5 Suppl 6:O31.PubMed CentralGoogle Scholar
  23. Vázquez-Vázquez M, Santana-López V, Skodova M, Ferrero-Álvarez-Rementeria J, Torres-Olivera A. Hand hygiene training through a serious game: new ways of improving Safe Practices. 2011 IEEE 1st International Conference on Serious Games and Applications for Health (SeGAH). Braga: IEEE; 2011.Google Scholar
  24. Castro-Sánchez E, Charani E, Moore L, Gharbi M, Holmes A. “On call: antibiotics”- development and evaluation of a serious antimicrobial prescribing game for hospital care. In: Schouten B, Fedtke S, Schijven M, Vosmeer M, Gekker A, editors. Games for Health 2014: Proceedings of the 4th conference on gaming and playful interaction in healthcare. Wiesbaden: Springer Vieweg; 2014. p. 1–7.Google Scholar
  25. Venier A-G, Marie S, Duroux T, Bervas C, Parneix P. Teaching good infection control practices with fun: impact of the serious game Flu.0. Antimicrob Resist. Infect Control. 2015;4 Suppl 1:I10.Google Scholar
  26. Gabbard JL, Hix D, Swan JE. User-centered design and evaluation of virtual environments. IEEE Comput Graph Appl. 1999;19:51–9.Google Scholar
  27. Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GE. Methods for the economic evaluation of health care programmes. 4th ed. Oxford: Oxford University Press; 2015.Google Scholar
  28. Mosadeghi S, Reid MW, Martinez B, Rosen BT, Spiegel BM. Feasibility of an Immersive Virtual Reality Intervention for Hospitalized Patients: An Observational Cohort Study. JMIR Ment Health. 2016;3(2):e28. doi:https://doi.org/10.2196/mental.5801.PubMedPubMed CentralGoogle Scholar
  29. Pan JJ, Ahn W, Dargar S, et al. Graphic and haptic simulation for transvaginal cholecystectomy training in NOTES. J Biomed Inform. 2016;60:410–21. doi:https://doi.org/10.1016/j.jbi.2016.03.005.PubMedGoogle Scholar
  30. Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6:42.PubMedPubMed CentralGoogle Scholar
  31. Shepperd S, Lewin S, Straus S, et al. Can we systematically review studies that evaluate complex interventions? PLoS Med. 2009;6:e1000086.PubMedPubMed CentralGoogle Scholar

Copyright

© The Author(s). 2016

Advertisement