ICS Development

ICS is the product of a collaboration between the Central Coast Local Health District, NSW, International Foundation for Integrated Care, University of Newcastle, and Flinders Filters (Flinders University).

To develop the filter, a gold standard set of references was drawn from the reference lists of Handbook Integrated Care, as well as from key integrated care reports by organisations such as WHO and Kings’ Fund. These references were checked by members of our Expert Advisory Group for relevance to integrated care. They were then split into three sets:

  • A Term Identification Set (for identifying terms describing the concept)
  • A Filter Development Set (for testing the performance of each IC term individually and in combination)
  • A Filter Validation Set (for confirming the final level of performance of the filter and ensuring it stays consistent across sets).

ICS performance

Search performance was measured in terms of:

  • Recall: the number of relevant citations the search retrieves as a proportion of all relevant citations
  • Precision: the number of relevant citations the search retrieves as a proportion of all citations retrieved (relevant or not relevant).

Ideally searchers want high recall and high precision, while in actual fact these two factors work as a trade-off. High recall usually comes at the cost of precision and vice-versa.

Recall values for the ICS filters are shown in the following table.

Version % Recall in Filter Development Set
(n=213)		 % Recall in the Filter Validation Set
(n=214)
Broad ICS filter 88.26
95% CI [83.25-91.92]		 85.98
95% CI [80.69-90.00]
Narrow ICS filter 55.87
95% CI [49.15-62.37]		 59.81
95% CI [53.13-66.15]

Publications

Lewis S, Damarell RA, Tieman JJ, Trenerry C. Finding the Integrated Care Evidence Base in PubMed and Beyond: A Bibliometric Study of the Challenges. International Journal of Integrated Care. 2018;18(3):11. DOI: http://doi.org/10.5334/ijic.3975

Acknowledgements

The project was developed with input from the following groups and individuals:

Project Lead:

  • Dr Suzanne Lewis, Central Coast Local Health District, Australia

Expert Advisory Group:

  • Dr Nick Goodwin, International Foundation for Integrated Care, UK
  • Dr Viktoria Stein, International Foundation for Integrated Care, UK
  • Dr Ellen Nolte, London School of Hygiene & Tropical Medicine, UK
  • Dr Frances Cunningham, Menzies School of Health Research, Australia
  • Dr Hazel Dalton, University of Newcastle. Australia
  • Emeritus Professor Jeff Fuller, Flinders University, Australia
  • Professor David Perkins, University of Newcastle, Australia
  • Dr Rachel Sheather-Reid, Central Coast Local Health District, Australia
  • Professor Jackie Cumming, Victoria University, New Zealand
  • Professor Mirella Minkman, TIAS/Tilburg University, Netherlands
  • Dr Esther Suter, University of Calgary, Canada
  • Nuria Toro, World Health Organization, Switzerland

Flinders Filters

  • Associate Professor Jennifer Tieman
  • Ms Raechel Damarell
  • Ms Camilla Trenerry