Skip to main content

The efficacy of resilience training programs: a systematic review protocol

Abstract

Background

Resilience has been defined as the ability of individuals to manage and adapt to stress and life challenges. Training programs that develop and/or enhance resilience may have efficacy in improving health, well-being, and quality of life. Because patients with chronic conditions must reliably self-manage their health, strategies to bolster resilience in this population may be of particular value. The objectives of this systematic review are to synthesize the evidence of resilience training program efficacy in improving outcomes related to quality of life, self-efficacy and activation, and resilience and coping ability in: 1) diverse adult populations; and 2) patients with chronic conditions.

Methods/Design

We will conduct a systematic review of randomized controlled trials assessing the efficacy of any program designed to enhance resilience in adults that measure any outcome against any comparator. We will search multiple electronic databases, trial registries, bibliographies, and will contact authors and experts to identify studies. We will use systematic review software to independently and in duplicate screen reports and extract data. We will extract characteristics of the study populations, interventions, comparators, outcomes, and quality/risk of bias. Primary, patient reported outcomes will be categorized into domains of quality of life, self-efficacy, and resilience. Secondary outcomes will be considered based on findings of the review. We will attempt meta-analysis by pooling standardized mean differences and minimally important differences (MIDs), when possible. Planned trial subgroup analyses are: 1) studies of patients with chronic conditions; 2) studies with placebo controls; 3) studies with similar intervention characteristics; and 4) studies with common lengths of follow-up.

Discussion

This study is intended to accumulate the evidence for resilience training programs in improving quality of life, resilience, and self-efficacy for care management, particularly among adult patients with chronic conditions. Its findings will be valuable to policy-makers, funding agencies, clinicians, and patients seeking innovative and effective ways to achieve patient-centered care.

Trial registration

PROSPERO registration number: CRD42014007185.

Peer Review reports

Background

Resilience has been defined as ‘the process of negotiating, managing, and adapting to significant sources of stress or trauma’ [1] and as ‘the ability of an individual to adjust to adversity, maintain equilibrium … and continue to move on in a positive manner’ [2]. Resilient persons often exhibit characteristics of commitment, dynamism, humor in the face of adversity, patience, optimism, faith, and altruism [2]. Although perhaps traditionally thought of as an intrinsic, non-modifiable human attribute, evidence suggests that resilience may be both a ‘movable’ and teachable construct [3–6]. It also appears to be heavily influenced by external and modifiable social factors [1, 7]. International interest in resilience research has increased in recent years and a number of stakeholders are considering the role resilience plays in influencing health, well-being, and quality of life [1, 8, 9]. Because patient self-perception of health and well-being is an important predictor of many relevant outcomes [10, 11], improving personal resilience may offer new and effective approaches to care delivery [3, 12, 13].

In chronic conditions, patients must reliably self-manage their care. The ability of patients to carry out this work is affected by a number of inter-related personal patient characteristics. These include activation [14], self-efficacy [15], and resilience [13]. Identifying effective ways to promote the development of these characteristics in this population can improve coping ability and prepare patients for the long and unpredictable journey of living with a chronic condition [16–19].

Training programs that develop and/or enhance resilience vary in structure and format and have been tried in a variety of populations. Mediating variables that are commonly addressed in these programs include the promotion of positive emotions, cognitive flexibility, social support, life meaning, and active coping [20]. Unlike mindfulness-based stress reduction (MBSR) or other meditation-based programs, which primarily seek to address active issues of stress and/or anxiety [21], resilience-specific training (as currently conceptualized) is primarily forward-looking (for example ‘preventive’) [3] and seeks to foster personal qualities needed to deal with unanticipated stressors. To our knowledge, no systematic review of the efficacy of resilience training programs exists. To better understand the potential for resilience training to bolster patient self-management capacity in chronic conditions, we will examine the literature on this topic.

The objectives of this proposed systematic review are to:

  1. 1)

    Synthesize the evidence of resilience training program efficacy in improving outcomes related to quality of life, self-efficacy and activation, and resilience and coping ability in diverse adult populations.

  2. 2)

    Estimate the aggregate efficacy of resilience training programs in improving quality of life, self-efficacy and activation, and resilience and coping ability among patients with chronic conditions.

Methods/Design

Study design

We will conduct a systematic review adhering to the reporting guidelines of the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement [22].

Study registration

This systematic review is registered with PROSPERO (registration number: CRD42014007185; http://www.crd.york.ac.uk/PROSPERO).

Criteria for considering studies for this review

Type of studies

We will include randomized controlled trials published within or after the year 1990 that evaluate the efficacy of any program specifically designed to develop or enhance subject resilience. We will exclude historically controlled, quasi-experimental, and single-arm pre-post studies.

Type of participants

We will include studies of all adults (≥18 years old). Studies of children will be excluded.

Type of interventions

Eligible interventions will include any program specifically designed to develop or enhance subject resilience, as determined by a consensus of the authors. These programs may have additional elements (MBSR, meditation, yoga, for example) but cannot be solely focused on stress-reduction or ‘reactive’ methods. Similarly, although many interventions are likely to affect resilience secondarily, we will restrict our inclusion only to those interventions that prospectively and systematically aim to impact this construct primarily. Such interventions should generally describe a theoretical or scientific rationale for why the intervention would be expected to impact resilience.

Type of outcome measures

We are primarily interested in patient reported outcomes that measure one or more of the following three construct domains: 1) quality of life or well-being; 2) activation or self-efficacy; and 3) resilience or ability to cope. Other secondary outcomes will be considered, depending on what is uncovered during the review; anticipated possibilities include depression, stress, and anxiety.

Search methods for the identification of studies

We will design and conduct a search strategy using methods recommended by the Institute of Medicine [23]. This will involve the searching of several electronic databases: PubMed, Scopus, EBSCO CINAHL, Ovid MEDLINE, Ovid EMBASE, Ovid Cochrane Library, Web of Science, and Ovid PsycINFO. Because resilience itself is an evolving construct [3, 24, 25] and the concept of resilience training is still relatively new, being continuously informed by scientific advances, and unlikely to have been operationalized in similar ways many years ago [3], databases will be searched only from January 1, 1990 to the current time. To maximize the specificity and feasibility of the search, we will focus on Medical Subject Headings (MeSH) terms such as ‘resilience, psychological’ and ‘randomized controlled trials’ along with text searches of ‘resilience’ and ‘hardiness’ (which is a term used by some in referencing a similar construct), among others. The concept of training will be addressed primarily using textwords such as ‘train,’ ‘teach,’ and ‘skills.’ The initial electronic search strategy will be supplemented by hand searching the reference lists from eligible included studies and through contacting experts in the field to identify any missing, in-progress, or unpublished studies. In addition, we will search for reviews on the topic and search through their reference lists to identify any potentially eligible studies that may have been missed through other methods. Finally, clinical trial registries will be searched to identify completed and in-progress studies and, if not identified through other methods, the authors will be contacted for details regarding the study’s status. There will be no language restrictions during the initial search.

Selection of studies

We will upload search results into systematic review software (DistillerSR, Ottawa, ON, Canada). In the first round of screening, abstracts and titles will be screened for inclusion. Following abstract screening, eligibility will be assessed through full-text screening. Prior to both abstract and full-text screening, reviewers will undergo training to ensure a basic understanding of the background of the field and purpose of the review. Comprehension of the inclusion and exclusion criteria will also be assessed through calibration on a small number of studies. Eligibility at both levels (abstract and full-text) will be assessed independently and in duplicate. Any disagreements will be resolved by consensus. If consensus cannot be achieved between the two reviewers, a third reviewer will arbitrate.

Data extraction

Data extractors, working independently, will collect primary data from the included trials through use of a web-based program (DistillerSR). The extracted data will include patient characteristics, outcomes measured and instruments used, intervention and control characteristics, and factors associated with study quality. Discrepancies in data collection will be adjudicated by consensus.

If data presented in the studies is unclear, missing, or presented in a form that is either un-extractable or difficult to reliably extract, the authors of the study will be contacted for clarification. When data extraction is complete, the authors of the studies will be contacted to ensure the accuracy and completeness of the data extraction. In addition, at this time the authors of included studies will be asked if they know of any additional studies, either completed or ongoing, that they believe would be eligible for our review.

Author contact will be initiated by an email to the corresponding author. If an email is unavailable an internet search will be used to find a current email address; when emails are available, first authors of manuscripts will be carbon copied on all emails to the corresponding author. If emails for the corresponding author are unavailable, corresponding authors will be contacted by telephone. Authors will be given a week to respond to emails, at which time a follow-up email will be sent, if no response is received after an additional two weeks a telephone call will be made to try to contact the author. Attempts to reach authors by telephone will occur throughout the week for a period of two weeks at which time the author will be classified as un-contactable.

Risk of bias assessment

We will use the Cochrane Collaboration’s risk of bias tool [26] to evaluate the methodological quality of included studies. The risk of bias (high/low/unclear) in included studies will be assessed in duplicate by reviewers working independently. Any disagreements will be resolved by consensus; if consensus is unable to be achieved a third reviewer will arbitrate. Items included in the risk of bias assessment will include: randomization, quality of randomization (any important imbalances at baseline), allocation concealment, level(s) of blinding/masking, losses to follow-up, intention to treat analysis, handling of missing data, and funding sources.

Analysis

We will first summarize and describe the populations, interventions, and outcomes studied. Descriptive statistics will be used as appropriate to compare the characteristics of the studies and narratives will be used as necessary to describe the interventions. The patient reported outcomes of the included studies will be categorized by construct similarity into the following domains: 1) quality of life or well-being; 2) activation or self-efficacy; and 3) resilience or ability to cope. This process will involve retrieval of the published description of each instrument’s construction and validation and/or a relevant review of instruments. Domain categorization will be based on consensus among the authors. To deal with the potential scenario in which multiple outcomes and/or measures are used within a single domain for a given study, we will develop an outcome hierarchy to determine which outcome to extract for analysis. This hierarchy will prioritize outcomes that are: a) commonly reported; b) highly validated and/or have a determined minimally important difference (MID); and c) endorsed by the National Quality Forum [27]. Additional, secondary outcomes will be considered based on what the review uncovers. For any part of this systematic review where subjective assessment is required, the agreement between two reviewers will be measured using kappa or phi statistics, as appropriate [28] (the latter is appropriate when the distribution of agreement is extreme).

Summary measures

We will conduct a meta-analysis of pooled standardized mean differences (SMDs) as suggested by the Cochrane Handbook for Systematic Reviews of Interventions[29] and additionally report, if possible, in terms of MIDs, as recommended by Johnston et al. [30, 31]. Alternatively, if the included studies consistently use a single, common instrument, we will consider converting outcomes from other instruments to the common instrument’s natural units and report the pooled effect in this way [30]. We will report both random and fixed effects models in the case of a discrepancy between them; otherwise, we will report the random effects model only. We will measure inconsistency for each outcome by estimating the I2 test and its associated confidence interval [32]. We will use RevMan software [33] to conduct the analyses.

Missing data

If missing data exists within the included trials, we will contact the authors to see if it is obtainable. If the data is unobtainable, we will use the complete case analysis and conduct sensitivity analysis for continuous outcomes and dichotomous outcomes using the methods of Ebrahim et al. [34] and Akl et al. [35], respectively.

Risk of bias across studies

Publication bias will be assessed by plotting the estimate of effect of trials by the inverse of its standard error using a funnel plot. The plots will be assessed both visually and by using Egger’s test [36]; a significant publication bias will be considered to exist if the P value is <0.1.

Additional analyses

Following the primary analyses, several a priori, exploratory sub-group analyses will be performed. These include stratification by: 1) whether the patient population had a chronic condition (defined as any condition expected to last at least one year and require ongoing management); 2) the type of comparator used (that is, placebo/sham versus usual care); 3) the intervention structure or format; 4) the length of follow-up; and 5) the quality of the study/risk of bias. We may perform additional subgroup analyses if the search reveals large numbers of studies in certain populations (for example, healthcare workers). If intervention structure proves to be highly variable and the number of studies is sufficiently large, we may attempt to explain variation in effects by conducting a meta-regression analysis of intervention characteristics.

Discussion

Healthcare should seek to prioritize outcomes that matter to patients [27]. In many cases, this may mean the pursuit of patient goals for life and health at the expense of metabolic parameters or guideline-based protocols. This subtle paradigm shift in care delivery requires a rethinking of the sorts of strategies that are most likely to be useful. Resilience training programs are an intriguing intervention that may have value in this regard, but no high-quality synthesis of the evidence exists.

This study will face a number of limitations that may limit its ability to generate conclusions based on high confidence. Specifically, there may be significant heterogeneity in the populations studied and the structure and theoretical approaches of interventions tried. There will also likely be differences in the outcomes measured and tools used. Pooling this data within construct domains carries inherent uncertainty.

We have also elected to exclude studies in children, which will limit the applicability of our findings to this population. Although many studies of resilience have been conducted in children, these have largely sought to understand the antecedents and predictors of resiliency. The question we seek to answer, however, is more concerned with determining the usefulness of interventions to promote resiliency; we anticipate that such interventions, in clinical practice, will more frequently be tried in adults.

Conclusion

This study is intended to accumulate the evidence for resilience training programs in improving quality of life, resilience, and self-efficacy for care management, particularly among adult patients with chronic conditions. Its findings will be valuable to policy-makers, funding agencies, clinicians, and patients seeking innovative and effective ways to achieve patient-centered care.

Abbreviations

MBSR:

Mindfulness-based stress reduction

MeSH:

Medical Subject Headings

MID:

Minimally important difference

PRISMA:

Preferred reporting items for systematic reviews and meta-analyses

SMD:

Standardized mean difference.

References

  1. Windle G, Bennett KM, Noyes J: A methodological review of resilience measurement scales. Health Qual Life Outcomes. 2011, 9: 8-10.1186/1477-7525-9-8.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Jackson D, Firtko A, Edenborough M: Personal resilience as a strategy for surviving and thriving in the face of workplace adversity: a literature review. J Adv Nurs. 2007, 60: 1-9. 10.1111/j.1365-2648.2007.04412.x.

    Article  PubMed  Google Scholar 

  3. Connor KM, Zhang W: Recent advances in the understanding and treatment of anxiety disorders: resilience: determinants, measurement, and treatment responsiveness. CNS Spectr. 2006, 11: 5-12.

    PubMed  Google Scholar 

  4. Waite PJ, Richardson GE: Determining the efficacy of resiliency training in the work site. J Allied Health. 2004, 33: 178-183.

    PubMed  Google Scholar 

  5. Steinhardt M, Dolbier C: Evaluation of a resilience intervention to enhance coping strategies and protective factors and decrease symptomatology. J Am Coll Health. 2008, 56: 445-453. 10.3200/JACH.56.44.445-454.

    Article  PubMed  Google Scholar 

  6. Maddi SR, Harvey RH, Khoshaba DM, Lu JL, Persico M, Brow M: The personality construct of hardiness, III: relationships with repression, innovativeness, authoritarianism, and performance. J Pers. 2006, 74: 575-597. 10.1111/j.1467-6494.2006.00385.x.

    Article  PubMed  Google Scholar 

  7. van Kessel G: The ability of older people to overcome adversity: a review of the resilience concept. Geriatr Nurs. 2013, 34: 122-127. 10.1016/j.gerinurse.2012.12.011.

    Article  PubMed  Google Scholar 

  8. Windle G: The contribution of resilience to healthy ageing. Perspect Public Health. 2012, 132: 159-160.

    Article  PubMed  Google Scholar 

  9. Almedom AM: Resilience research and policy/practice discourse in health, social, behavioral, and environmental sciences over the last ten years. Afr Health Sci. 2008, 8 (Suppl 1): S5-S13.

    PubMed  PubMed Central  Google Scholar 

  10. DeSalvo KB, Bloser N, Reynolds K, He J, Muntner P: Mortality prediction with a single general self-rated health question. J Gen Intern Med. 2006, 21: 267-275. 10.1111/j.1525-1497.2005.00291.x.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Wong FK, Chan MF, Chow S, Chang K, Chung L, Lee WM, Lee R: What accounts for hospital readmission?. J Clin Nurs. 2010, 19: 3334-3346. 10.1111/j.1365-2702.2010.03366.x.

    Article  PubMed  Google Scholar 

  12. Jeste DV, Savla GN, Thompson WK, Vahia IV, Glorioso DK, Martin AS, Palmer BW, Rock D, Golshan S, Kraemer HC, Depp CA: Association between older age and more successful aging: critical role of resilience and depression. Am J Psychiatry. 2013, 170: 188-196. 10.1176/appi.ajp.2012.12030386.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Franco OH, Karnik K, Osborne G, Ordovas JM, Catt M, van der Ouderaa F: Changing course in ageing research: the healthy ageing phenotype. Maturitas. 2009, 63: 13-19. 10.1016/j.maturitas.2009.02.006.

    Article  PubMed  Google Scholar 

  14. Greene J, Hibbard JH: Why does patient activation matter? An examination of the relationships between patient activation and health-related outcomes. J Gen Intern Med. 2012, 27: 520-526. 10.1007/s11606-011-1931-2.

    Article  PubMed  Google Scholar 

  15. Romppel M, Herrmann-Lingen C, Wachter R, Edelmann F, Dungen HD, Pieske B, Grande G: A short form of the General Self-Efficacy Scale (GSE-6): development, psychometric properties and validity in an intercultural non-clinical sample and a sample of patients at risk for heart failure. Psychosoc Med. 2013, 10: Doc01-

    PubMed  PubMed Central  Google Scholar 

  16. Hilliard ME, Harris MA, Weissberg-Benchell J: Diabetes resilience: a model of risk and protection in type 1 diabetes. Curr Diab Rep. 2012, 12: 739-748. 10.1007/s11892-012-0314-3.

    Article  PubMed  Google Scholar 

  17. Edward SL: Chronic illness and wellbeing: using nursing practice to foster resilience as resistance. Br J Nurs. 2013, 22: 741-742. 744, 746

    Article  PubMed  Google Scholar 

  18. Ma LC, Chang HJ, Liu YM, Hsieh HL, Lo L, Lin MY, Lu KC: The relationship between health-promoting behaviors and resilience in patients with chronic kidney disease. Sci World J. 2013, 2013: 124973-

    Google Scholar 

  19. Vitali S: Finding quality of life despite MS: harnessing resilience. Int MS J. 2011, 17: 94-99.

    PubMed  Google Scholar 

  20. Sturgeon JA, Zautra AJ: Resilience: a new paradigm for adaptation to chronic pain. Curr Pain Headache Rep. 2010, 14: 105-112. 10.1007/s11916-010-0095-9.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Goyal M, Singh S, Sibinga EM, Gould NF, Rowland-Seymour A, Sharma R, Berger Z, Sleicher D, Maron DD, Shihab HM, Ranasinghe PD, Linn S, Saha S, Bass EB, Haythornthwaite JA: Meditation programs for psychological stress and well-being: a systematic review and meta-analysis. JAMA Intern Med. 2014, doi:10.1001/jamainternmed.2013.13018

    Google Scholar 

  22. Moher D, Liberati A, Tetzlaff J, Altman DG: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009, 6: e1000097-10.1371/journal.pmed.1000097.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Institute of Medicine: Finding What Works in Health Care: Standards for Systematic Reviews. 2011, Washington, DC: The National Academies Press

    Google Scholar 

  24. Earvolino-Ramirez M: Resilience: a concept analysis. Nurs Forum. 2007, 42: 73-82. 10.1111/j.1744-6198.2007.00070.x.

    Article  PubMed  Google Scholar 

  25. Windle G, Markland DA, Woods RT: Examination of a theoretical model of psychological resilience in older age. Aging Ment Health. 2008, 12: 285-292. 10.1080/13607860802120763.

    Article  PubMed  Google Scholar 

  26. Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA: The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011, 343: d5928-10.1136/bmj.d5928.

    Article  PubMed  PubMed Central  Google Scholar 

  27. National Quality Forum: Patient Reported Outcomes (PROs) in Performance Management. 2013, Washington, DC: National Quality Forum

    Google Scholar 

  28. Guyatt G, Rennie D, Meade M: Users’ Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. 2008, McGraw-Hill Education: Maidenhead, 2

    Google Scholar 

  29. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0. Edited by: Higgins JP, Green S. 2011, Oxford: The Cochrane Collaboration

    Google Scholar 

  30. Johnston BC, Patrick DL, Thorlund K, Busse JW, da Costa BR, Schunemann HJ, Guyatt GH: Patient-reported outcomes in meta-analyses - part 2: methods for improving interpretability for decision-makers. Health Qual Life Outcomes. 2013, 11: 211-10.1186/1477-7525-11-211.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Johnston BC, Thorlund K, Schunemann HJ, Xie F, Murad MH, Montori VM, Guyatt GH: Improving the interpretation of quality of life evidence in meta-analyses: the application of minimal important difference units. Health Qual Life Outcomes. 2010, 8: 116-10.1186/1477-7525-8-116.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Higgins JP, Thompson SG, Deeks JJ, Altman DG: Measuring inconsistency in meta-analyses. BMJ. 2003, 327: 557-560. 10.1136/bmj.327.7414.557.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Centre TNC: Review Manager (RevMan). Version 5.2. 2008, Oxford: The Cochrane Collaboration

    Google Scholar 

  34. Ebrahim S, Akl EA, Mustafa RA, Sun X, Walter SD, Heels-Ansdell D, Alonso-Coello P, Johnston BC, Guyatt GH: Addressing continuous data for participants excluded from trial analysis: a guide for systematic reviewers. J Clin Epidemiol. 2013, 66: 1014-1021. 10.1016/j.jclinepi.2013.03.014.

    Article  PubMed  Google Scholar 

  35. Akl EA, Johnston BC, Alonso-Coello P, Neumann I, Ebrahim S, Briel M, Cook DJ, Guyatt GH: Addressing dichotomous data for participants excluded from trial analysis: a guide for systematic reviewers. PLoS One. 2013, 8: e57132-10.1371/journal.pone.0057132.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Egger M, Davey Smith G, Schneider M, Minder C: Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997, 315: 629-634. 10.1136/bmj.315.7109.629.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

This publication was made possible by CTSA Grant Number UL1 TR000135 from the National Center for Advancing Translational Sciences (NCATS), a component of the National Institutes of Health (NIH). Its contents are solely the responsibility of the authors and do not necessarily represent the official view of the NIH. The funding source did not participate in any part of this study.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Aaron L Leppin.

Additional information

Competing interests

One of the authors, AS, has published extensively on this topic and has written a book about resilience. AS did not conceptualize this project or any part of its approach, but was sought out for expertise and direction.

Authors’ contributions

AL, JT, VM, AS, PE, CZ, BR, KB, MG, PB, MD, and JB conceived and designed the study, and wrote the manuscript. All authors read and approved the final manuscript.

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article

Leppin, A.L., Gionfriddo, M.R., Sood, A. et al. The efficacy of resilience training programs: a systematic review protocol. Syst Rev 3, 20 (2014). https://doi.org/10.1186/2046-4053-3-20

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/2046-4053-3-20

Keywords