Advancing integrated care evaluation in shifting contexts

Authors: Jenny Billings , Simone R. de Bruin , Caroline Baan and Giel Nijpels

Despite many studies evaluating the effectiveness of integrated care, evidence remains inconsistent. Increasingly, commentary on the subject of integrated care evaluation is pointing out the mismatch between the ability to capture the impact of integrated care initiatives and programmes, and the selection of the most appropriate methodology to do this. Authors have highlighted a range of evaluation challenges that include the stability and sustainability of initiatives; data collection and the suitability of measures; and a lack of appreciation of the complexity. In addition, the status of integrated care as a ‘process’ must be recognised, meaning it is not a ‘fixed’ intervention, but susceptible to constant development and change. These factors all affect the sturdiness of evaluation designs and what constitutes an outcome. This in turn is prompting the need to fit the evaluation design more with how integrated care is implemented in practice and what integrated care is there to achieve and improve. The value of mixed methods studies and suitable frameworks that examine both processes and outcomes has therefore been recognised in this field.

Regarding the examination of processes, the wide variation in how integrated care is operationalised calls for evaluations that include a range of qualitative methods, so that important contextual information can be examined to identify what seems to work and why. Regarding outcomes, there is a need to ensure that outcome measures have a good pragmatic fit with the shifting context of integrated care interventions and the population group under study. There is a tendency for example for measures for frail older people and people with multimorbidity to focus on general health outcomes (e.g. health status, physical functioning, quality of life), which may not be appropriate to their fluctuating physical and mental status. Outcomes such as experiences of care, independence and autonomy may be more suited to this vulnerable target group, and these inclusions may be more appropriate to ascertain the link between the integrated care processes and what improvements can be expected for the service user in receipt of care.

Given this impetus, researchers are adopting more ‘real world’ methodologies for the evaluation of complex interventions such as integrated care. While mixed methodologies have been advocated for some time to gain a better appreciation of the ‘grass root’ processes involved in integrated care implementation, the emergence of realist approaches drawn from Pawson and Tilley (1997)  has become evident. Realist researchers seek to explain the underlying “cause” or mechanisms that generate observed phenomena through the construction of context, mechanism and outcome (CMO) configurations. To support this, academics are developing frameworks such as the COMIC model for the evaluation of integrated care. In addition to realist methods, and continuing with the focus on context, researchers are turning to implementation research. This is described as the scientific study of the processes used in the implementation of an initiative alongside the context within which it is taking place. Its intention is to promote the systematic uptake of research findings and other evidence-based practices into routine practice, and, hence, to improve the quality and effectiveness of health services and care.

This broadening of methods appeal is becoming reflected in funding opportunities. The European Union (EU) funded research initiatives (Horizon 2020) are encouraging the adoption of innovative approaches to the evaluation of integrated care to better understand the impact on vulnerable populations with complex needs. One such project is SUSTAIN – Sustainable Tailored Integrated Care for older people in Europe (2015–2019). This paper focuses on the use of innovative approaches within SUSTAIN, and describes the method by which implementation research and case study design were blended for the evaluation of European integrated care initiatives. The aim is to share methodological experiences and lessons learned with the research community, in order to advance understanding of integrated care evaluation in context and add to the international ‘toolbox’ of methodological approaches. It will commence with a brief introduction to SUSTAIN and an overview of the design. This is followed by a critical discussion of the strengths and weaknesses of our approach and concludes with an assessment of the extent to which we advanced understanding of integrated care evaluation. Lessons learned and recommendations for future integrated care evaluations are put forward.