Holly Dorning, Research Analyst, The Nuffield Trust - photo

Holly Dorning

Research Analyst
Nuffield Trust

At the end of 2014, Solutions for Public Health (SPH) was commissioned to conduct a study to gather consensus from a range of stakeholder groups on what they felt was important to measure in order to understand the quality of healthcare. This study used a Delphi-style consultation approach to interrogate 73 statements that describe the measurable aspects of the quality of healthcare. This work was part of the QualityWatch programme and concluded in February 2015. This is a summary of the findings of the work. It also provides further context and highlights the impact of the results.

Key findings

  • All 73 statements were rated very highly on the scale of importance. Over 90 per cent of the statements were scored as ‘very important’ or ‘important’. Average scores (on a scale of 1 to 5) for all statements ranged from the lowest score of 3.89 to the highest score of 4.90.
  • Statements that related to safety scored relatively highly and the domain contained six of the ten highest scoring statements. In contrast to safety, the domain that related to access to care contained four of the lowest scoring statements.
  • Broadly, there was little variation in how the different stakeholder groups scored the statements. Over 60 per cent of the scores given by each respondent group fell within the ‘very important’ category.
  • As no statements came out as scoring particularly low on the importance scale, it highlights the need to continue to use many different measures to understand the quality of care. It also suggests that using indicators is a good approach for covering the ‘many things to many people’ nature of quality.

1. The challenges of measuring quality

‘Quality’ is a complex concept to define (QualityWatch, 2013). Often, the definition is dependent on individual perspective and on the context of where care is being given. This subjective nature indicates that quality can mean many different things to different people, in different situations (McGlynn, 1997).

Many frameworks have been used to categorise measures of quality in order to capture these different perspectives and settings. The most famous one is probably that used by Donabedian (1997), which groups statements into three categories:

  • structure (e.g. looking at the number of doctors)
  • process (which looks at what happened to the patient when they received care, such as what treatment they had and how long they waited)
  • outcomes (which looks at the effect this care had on the patient, for example whether the treatment led to improved health).

This approach has evolved over time and QualityWatch uses a framework, drawing on work by Leatherman and Sutherland (2008), which views quality through six domains and by broad health or care sector. Indicators that measure these slices of quality then fit within this framework. Figure 1 shows the QualityWatch framework and how many measures or ‘indicators’ exist in each group. This is based on the data available and is not reflective of the selection process made by the QualityWatch programme.

 

To date, there has been little systematic work done to find out whether people feel that certain measures have more value in understanding the quality of healthcare than others. Adding to this, different people – including professionals, care users, their friends and relatives and policy-makers/influencers – may have different and conflicting opinions on what should be measured in order to understand quality.

Some research studies have been carried out aiming to gather the comparative views of patients and professionals on more focused areas. For example, some studies have found agreement within a particular subject area, such as heart failure (Browne and others, 2014) or the quality of women’s healthcare (Stichler and Weiss, 2000). However, there has been limited work looking at the much wider definition of healthcare quality and we felt that there would be little consensus between different stakeholders (Brown, 2007).

Therefore, the focus of this study was to understand the relative value of different quality measures and to seek a consensus from a range of different groups – from patients and carers, to healthcare professionals and policy-makers – on what they felt was important to measure in terms of understanding the quality of healthcare.

2. Gauging consensus

At the outset of this study, the QualityWatch programme had internally catalogued over 500 indicators within the matrix outlined in Figure 1 and had externally published analysis covering more than 200 of these. For the study, a pragmatic approach was therefore taken to only consider measures already available, rather than asking participants to suggest new quality measures from which to prioritise.

Due to the large number of indicators and uneven spread across the sectors/domains (see Figure 1), in the first stage of the study these indicators were organised into ‘themes’. Each theme spanned numerous indicators and could be turned into value statements about quality. From the list of over 500 indicators, 73 statements were derived. The themes were similar in style to National Voices’ ‘I statements’ (National Voices, 2013). An example theme is ‘avoidable admissions to hospital’ and the corresponding value statement is: ‘patients receive enough care and support to avoid unnecessary hospital admissions’. For a full list of the value statements, see Appendix 1: Value statements.

This study used a Delphi-style consultation approach to interrogate the statements. Delphi methodologies form part of a wider set of consensus methodologies and are used to gather the opinions of various groups on issues that are complex and multifaceted. The approach usually consists of three rounds. The first round is designed to explore the area under discussion. The second round feeds back on the responses of the first and introduces a further set of questions, which may seek to refine previous responses or to further explore the ideas raised. The final round focuses on consensus on the outcome (Linstone and Turoff, 2002).

In this study, the first round used a mixed-methods approach to capture a range of voices within the different stakeholder groups and included:

  • an online survey of 186 health and social care professionals, policy-makers and researchers
  • eight theme-based focus groups with 59 patients, service users and carers.

Focusing on each domain at a time, the online survey and focus groups covered similar questions, asking participants to:

  • rate (on a five-point scale) how important each statement was in understanding the quality of care that people receive
  • comment on whether the right statements were included in each domain
  • highlight any statements they thought were missing in understanding the quality of care in each domain.

The online survey asked participants to look at all 73 statements. However, due to the large number, each focus group covered statements in only two to three domains. Using the five-point scale (from 1 = not at all important, to 5 = very important), a mean score could be given to each statement to allow for comparisons.

The second and third rounds of the consensus process involved convening an online panel, made up of 42 mixed participants from the first round. Of the 42, 22 people responded to round two and seven people responded to round three.

In the second round, the panel was sent key findings from the first round and asked to comment on them in an online survey, reflect on whether their views supported the scores and try to explain any differences. This included reflecting on the highest and lowest scoring statements and those with the highest variation. The purpose of this round was to seek agreement from the panel members or explain discrepancies. In the third round, a summary of the panel’s comments from the second round was shared with all panel members and any final comments were invited.

3. Which aspects of quality were most important?

All 73 statements were rated very highly on the scale of importance by all respondents. For all domains, over 90 per cent of responses fell within the ‘very important’ or ‘important’ categories (see Figure 2). Average scores for all statements ranged from the lowest score of 3.89 to the highest score of 4.90. This shows that even the lowest scoring statements were rated as a ‘3’ (moderately important), reflecting the high scoring seen in the study overall. Figure 2 also shows how the distribution of scores varied by domain, with the highest number of ‘very important’/‘important’ ratings in the safety domain; and the lowest number in the access and equity domains.

 

Safety

Patients and service users should not be harmed by the care they receive or be exposed to unnecessary risk.

Safety was the highest scoring domain overall and contained six of the ten highest scoring statements (see Figure 3). The highest scoring statements related to patients not experiencing additional harm as a result of needing care. The statement ‘Patients don’t suffer abuse or injury while receiving care’ had the highest mean score (4.90) of all statements in the study. The importance of the safety domain was confirmed in round two of the study, with one respondent concluding the following:

“I would suggest that safety is the objective that everything is leading to. In order to ensure that patients are not harmed, appropriate systems, processes, procedures and culture need to be in place.” (Voluntary sector organisation focusing on patient and public involvement)

 

Person-centred care

Care should be based on partnerships between professionals and patients (and, where appropriate, their families). It should be delivered with compassion, empathy and responsiveness to the needs, values and preferences of the individual service user.

Statements relating to person-centred care had more variable scores compared with those in the safety domain. The proportion of respondents rating statements as ‘important’ or ‘very important’ ranged from 76 per cent to 99 per cent (see Figure 4). The highest scoring statements related to patients feeling valued and respected. The second round of the study led some respondents to highlight that the feeling of value and respect should extend beyond just the patient:

“Patients and their loved ones should feel respected, be listened to and have dignity. The ‘being listened to’ element is very important.” (Allied health, policy adviser)

Two statements in particular scored relatively lower than the rest. The first was: ‘places of care are recommended’. Comments revealed that people felt that recommendations were less valuable as patients do not have genuine choice as to where they receive care. It was also felt that recommendations will be subjective, with different patients wanting and needing different things.

The second low scoring statement was: ‘places of care don’t have high levels of upheld complaints’. Comments from rounds one and two of the study suggest that people felt that it was important to gather information relating to complaints but that simply counting complaints was not enough. This suggests one area where the indicator/measure could benefit from change:

“Focus should be on how the complaints are dealt with, rather than how frequently they occur. If residents in a care home, for example, are able to complain, feel confident in speaking up, then that is a positive – a large number of complaints don’t automatically mean that it is a bad care home, only if it’s continually high. Mistakes can be learned from.” (Focus group respondent)

 

Equity

Care should be provided on the basis of need, regardless of personal characteristics such as age, gender or ethnicity. It should be delivered in such a way as to reduce differences in health and wellbeing status and outcomes across sub-groups.

Over 80 per cent of respondents scored all statements in the equity domain as ‘important’ or ‘very important’ (see Figure 5). Two statements had lower scores, with a higher proportion of respondents feeling they were ‘moderately important’ or ‘of little importance’ compared with others. These statements related to the presence of something such as ‘services support people who speak a different language’ (this was framed in the value statement as patients being offered a translation service) and ‘staff are trained on equality and diversity’. Many of the comments in this area sought to extend the thinking behind these statements – that we should measure not just whether or not something exists (such as training) but also how good/effective it is:

“The quality of a service is more important and will reflect whether or not training has not just been of high quality in itself but also taken on board by the staff trained. It is the attitudes of staff towards providing quality and individualised service to all patients that will determine how well served (or not) patients (and their family carers) are cared for.” (Patient and public involvement lead)

 

Effectiveness

Services should be based, as far as possible, on relevant rigorous science and research, and/or nationally agreed best practice.

Over 80 per cent of respondents scored all statements in the effectiveness domain as ‘important’ or ‘very important’ (see Figure 6). Some of the highest scoring statements in this domain related to the organisation of care. One, ‘Operations are performed by experienced staff’ scored particularly highly. Many respondents felt that this theme was self-evident but added that competence goes hand in hand with experience:

“I would add that occupational or ‘real time’ competence is also critical – experience does not necessarily equal good or effective care.” (Social care manager)

One statement that scored relatively low was: ‘GP surgeries have good clinical performance’. Further discussion around this statement revealed that performance in this area should be monitored but that performance alone would not provide a measure of the quality of GP surgeries. It was also felt that the make-up of a local population would impact on this measure:

“Variation in clinical conditions is potentially influenced by geography, leading to some GPs seeing more of a particular condition and enhancing their skill in diagnosing/treating this. This should not allow deterioration on other measures of quality to be acceptable.” (Occupational therapist)

 

Capacity

Care systems should be sufficiently well resourced and with adequate distribution to enable delivery of appropriate services.

There was much greater variation in the scoring of statements in the capacity domain, particularly in the proportion of respondents feeling that statements were ‘very important’. This ranged from 30 per cent to 82 per cent (see Figure 7). Two of the highest scoring statements related to there being enough staff time to care for patients: ‘there are enough staff to care for patients’ and ‘staff do not experience work pressure or stress’.

The statement that generated the greatest variation and most debate was: ‘care is provided by the same staff’; just 30 per cent of respondents felt that this statement was ‘very important’. Some respondents felt that having the same staff was essential for developing the staff–patient relationship but that the importance varied depending on the care setting:

“Not pleasant for patient and carer to have multiple different people coming through the house – an extra thing that the carer has to think about, e.g. making sure they have the right information. It may be that a patient, according to ‘the book’, is fine because they have a wide range of support and are receiving care – but … if staff don’t have access to the same information there is a risk of things getting missed.” (Focus group respondent)

Some acknowledged that, in certain situations, waiting for a specific staff member came with a disadvantage (such as the waiting time for a GP appointment) and that, for some situations, many different staff members with varying expertise would be needed for effective care. Overall, it was felt that continuity of care was very important but that did not necessarily mean the same staff:

“Continuity of care is the key, not the staff necessarily. This is where good effective communication is a key area.” (Service user)

 

Access

Care services should be timely and provided within the appropriate setting, with access to necessary skills and expertise.

In contrast to other domains, the access domain (see Figure 8) contained four of the lowest scoring statements. It was generally thought that:

“The collection of statements [in the access domain] felt much less objective and less ‘real’ than those in the safety domain and addressed aspects of quality that people could reasonably expect e.g. ‘patients are admitted to a bed on a ward or unit promptly’”. (Focus group respondent)

‘GP surgeries offer a wide range of services’ was the lowest scoring statement in the study. It was felt that this was because the importance here should be placed on GP surgeries doing things well with appropriately trained staff rather than offering a wide range of services. Respondents also highlighted that GP surgeries should signpost and refer people to other services rather than offering a wide range of services themselves.

The highest scoring statement ‘people can access care in an emergency’ was felt to be an essential part of high-quality care:

“24/7 emergency access and clarity is essential to high-quality care.” (Associate medical director)

 

4. Did health and care professionals and patients agree?

Broadly, there was little variation in how the different stakeholder groups scored the statements overall. Over 60 per cent of the scores given by each stakeholder group fell within the ‘very important’ category. However, if we look at the domains, there were some differences between what patients and carers felt was important to measure and what clinicians felt was important (see Figure 9).

Overall, statements about safety, access and effectiveness were scored similarly by patients and carers and professionals.

Conversely, statements related to person-centred care and capacity were seen to be less important for patients and carers than for professionals. For person-centred care, 87 per cent of statements were scored as ‘important’ or ‘very important’ by patients and carers compared with 96 per cent of statements by clinicians. There were even more pronounced differences for statements in the capacity domain (77 per cent for patients and carers compared with 94 per cent for clinicians). The difference here was largely explained by patients and carers rating three particular statements of low importance: ‘care is provided by the same staff’, ‘staff have the relevant qualifications’ and ‘staff are well supported to do their job’.

The only domain in which clinicians overall felt that statements were less important than patients and carers was equity. As discussed previously, this may be due to people feeling that some of the statements in the domain slightly ‘miss the point’ as they measure the presence of something such as a translation service rather than the quality of that service.

 

5. Other findings

As well as the specific findings for each domain, respondents often commented that some of the statements, although covering important areas, were not quite measuring the right thing or did not go far enough to understand the quality of care. For example, for statements that covered whether patients received enough information, respondents felt that it wasn’t just about receiving enough information – it was important to also capture whether people understood the information and had places to go for further support.

People also thought that there were some gaps in the settings covered by the statements. It was generally felt that the statements focused more on the acute sector and were not very comprehensive in understanding the boundaries of care – such as the interaction between the acute sector and social care.

Respondents also believed that the statements should not be measured in isolation as context is key to understanding quality. For example, for the statement on the continuity of care from the same staff, they highlighted that continuity was important to a quality service in some settings – such as social care, where staff are entering someone’s home. However, in other settings, such as the acute sector, patients would expect to see a range of professionals with different expertise.

6. What does this mean for quality measurement?

At the onset of this study, we believed that different stakeholders would value quality measurements differently and that a consensus method would be needed to gain agreement between patient/carer groups and professionals in particular. However, the study showed that there was broad agreement and that generally what patients and carers felt it was important to measure to understand the quality of care was well aligned with the opinions of professionals. This is an encouraging finding and should be taken as a positive endorsement of the range of quality indicators in general and of the QualityWatch programme. Where there was more variation in scores, this was often due to the measure ‘missing the point’ rather than the theme of quality not being considered important.

While this study sought to gain consensus from varying groups, given the high level of consensus in the study, on reflection a different methodology may have been more appropriate. Forcing people to choose which statement is more important relative to other statements may have provided more meaningful results and told us more about which areas are more important to measure in understanding quality. An element of this was attempted in rounds two and three but due to the high number of statements it was not the primary objective.

Some of the findings do suggest that changes in what is measured would be beneficial. For example, just measuring whether a service exists (such as a translation service) tells us little about the quality of that service.

Looking at how each domain was scored, it is interesting to note that access was scored as comparatively less important than the other domains – yet this area has some of the most well-known measures and has also come to be one of the most easily measured aspects of quality (particularly in the acute sector). Our findings support work done by others, which suggests that undue focus on access targets (such as the four-hour Accident & Emergency target) does not necessarily capture quality in a way that patients and staff perceive to be most important (Blunt and others, 2015).

Conversely, it is not surprising that statements which relate to safety were seen to be very important in understanding the quality of care. However, this is an area where measurement is particularly challenging, given the nature of ‘safety’ and the lack of information in this area (QualityWatch, 2013).

It is interesting to note that statements in the person-centred care domain were considered much more important by professionals (96 per cent of statements were scored as ‘important’ or ‘very important’) compared with patients and carers (87 per cent). This finding suggests that the high importance placed on this domain by professionals is not matched by patients and raises questions about whether we are truly measuring what is valued by patients. Are the specific measures used not right or do patients simply feel that this broad area of measurement is not important?

In conclusion, we know that quality is complex and measuring it requires us to use many different indicators which aim to capture different settings and perspectives. These indicators have been tracked over many, many years, yet up until this study, little systematic research had been conducted to understand whether any of these measures are more important in understanding the quality of care than others, and whether different groups of people feel differently about this. Our findings suggest that, overall, the relatively high scores of importance given to the statements by all groups gives justification and validation to the indicators analysed in the QualityWatch programme. It also suggests that using a range of different indicators is a good approach to dealing with the multidimensional nature of quality. However, we should exercise caution to ensure that we understand the context around the indicators, and that we do not take measurement alone as a comment on quality.

As we continue to measure quality in this way, it is essential to keep questioning whether what we measure is important to those using and delivering health and care services. We must also ensure that the indicators provide value so that they are ultimately used to improve the care delivered.

Appendix 1: Value statements

Safety

  • Patients receive enough care and support to avoid unnecessary hospital admissions.
  • Patients don’t experience avoidable complications/harms from healthcare e.g. healthcare associated infection, medication errors, surgical errors.
  • Health and care organisations have appropriate procedures in place to deal with issues of safety or harm e.g. policies and training.
  • Healthcare staff work in an environment where they can report issues of safety or harm.
  • Healthcare staff work in a safe environment free from harassment, bullying and abuse.
  • After an appointment or hospital stay, patients are given enough information e.g. about possible side-effects of medication or danger signals to be aware of.
  • Places which provide care are clean.
  • Places which provide care have hand washing materials available for patients and visitors.
  • Healthcare staff wash their hands regularly e.g. between different patients.
  • Patients report feeling safe when receiving care e.g. during their hospital stay.
  • Patients don’t suffer abuse or injury while receiving care.
  • Hospitals don’t have a high, unexplained, death rate.

Person-centred care

  • Carers are given enough support to ensure they have a good quality of life.
  • Carers have enough information about services and are included in decisions about the person they care for.
  • Places which provide care do not have a high level of upheld complaints.
  • Places which provide care are able to respond to and deal appropriately with concerns and complaints.
  • Places where care is provided are recommended by patients.
  • Patients report a good overall experience of care.
  • Healthcare professionals know their patients’ medical history.
  • Healthcare professionals treat patients with dignity and respect.
  • Patients have a care plan in place following discharge from hospital.
  • Patients receive care in an environment which supports privacy and respect.
  • Patients feel listened to and involved in decisions about their care and treatment.
  • Patients are given enough information about their condition or treatment.
  • Healthcare staff support patients with everyday activities while they are receiving care e.g. help with eating or going to the toilet.
  • Where possible, people are able to die in a place of their choice.
  • Healthcare professionals don’t talk in front of patients as if they’re not there.
  • Patients receive enough support to manage their condition themselves.

Equity

  • Places which provide care are easily accessible for people with disabilities.
  • Places which provide care offer people with hearing problems the services they need, such as signing services or hearing loops.
  • Places which provide care offer translation services for people who don’t speak English.
  • Healthcare professionals have had equality and diversity training.
  • All healthcare staff are given equal opportunities for career progression and promotion.
  • People from different communities have the same health outcomes e.g. life expectancy.
  • People from different backgrounds and communities have the same access to care.

Effectiveness

  • People with long-term health problems are supported to live as full and independent lives as possible.
  • Everyone has the appropriate support and information to lead a healthy lifestyle e.g. smoking cessation and weight management.
  • GP surgeries score well on measures of clinical performance.
  • Hospital care adheres to good practice.
  • People who have mental health problems have the same health outcomes (e.g. mortality rates) as those who don’t.
  • People with long-term health problems (e.g. diabetes) get appropriate ongoing care and support.
  • The treatments patients have are successful and effective.
  • Local health services offer effective services for mothers, babies and children.
  • Healthcare is organised to avoid unnecessary hospital visits or patients staying in hospital too long.
  • Patients get enough support to avoid preventable health problems.
  • People who are at risk of suicide have access to interventions and support.
  • People with mental health needs also receive support for their physical health needs.
  • Health and care provides support which helps people live longer e.g. in cancer survival rates.
  • Procedures and operations are performed by people or hospitals with an appropriate level of experience.
  • Patients receive a diagnosis quickly enough to be beneficial for their health.

Capacity

  • There are enough staff to care for and support patients.
  • Patients receive continuity of care from the same staff.
  • Health and social care staff have relevant qualifications.
  • Organisations that provide care, support their staff to be capable and competent e.g. annual appraisals and personal development.
  • Healthcare staff have appropriate training.
  • Healthcare staff are not under too much pressure or stress to do their job well.

Access

  • Patients have enough time with their GP, nurse or health professional during their primary care appointment (at their GP surgery).
  • GP surgeries offer a wide range of services.
  • Patients can choose convenient appointment times to see healthcare professionals at their GP surgery and at hospitals.
  • Patients can easily join and access a GP surgery or dental service.
  • Ambulances arrive to patients promptly.
  • When patients are given a hospital appointment or operation date, the date and time don’t change or get cancelled.
  • When patients need an appointment they don’t have to wait too long.
  • When patients are ready to leave hospital they can do so easily and without unnecessary delays.
  • Patients are admitted to a bed on a ward or unit promptly.
  • Patients have access to appropriate care and support following a hospital visit.
  • Patients with mental health problems get appropriate care and support.
  • Patients with mental health problems are supported to live as full and independent lives as is possible.
  • People have access to health screening programmes which could lead to improved health.
  • People have access to vaccination programmes which could protect their health.
  • Patients who need specialist equipment in their home, are able to get it quickly.
  • People have access to home care/home help.
  • Patients can access care in an emergency.

About the researchers

Holly Dorning is a Research Analyst at the Nuffield Trust. She joined the Trust in May 2013 and has a particular interest in measuring changes in the quality of health and social care. Prior to joining the Trust, Holly worked for the Multiple Sclerosis (MS) Society, where she led the analysis of the ‘My MS, My Needs’ project, which explored whether people with MS were getting what they needed from health and social care.

Solutions for Public Health is an NHS public health consultancy. It works with decision makers across the public and third sectors to improve health and reduce health inequalities. It helps clients provide the right services in the right place, for the right population, at the best cost and quality.