Stage 2 – Implementing PREMs

Outcome

By completing step 2.1 you will have an overview of how you will establish routine use of the PREM in a way that suits the circumstances and resources of the organisation. A rollout strategy should outline the intended phases of implementation and a rationale for this. It needs to consider the need for cultural, technological, process and reporting changes to ensure the successful administration and use of the PREM survey and results.

Things to consider

This section lists the items that need to be considered in Step 2.1 to match your rollout strategy to your circumstances.

Variables affecting your rollout strategy

There are many variables to consider that may affect your rollout strategy:

• The time you have available will affect the number of stages and amount of formative evaluation you can carry out before full implementation is expected
• If you intend to amend the PREM, or to administer it differently or with patient populations that are different to populations that the PREM was tested with, this will affect the scale and analysis of your pilot testing; you may first need to establish the validity and reliability of the redesigned survey
• If you are adding the PREM to an existing survey, you may first need to establish the validity and reliability of the redesigned survey as there is potential for interference between the PREM and existing questions
• The type and features of existing patient experience survey programs will affect the level of technological change and staff training required
• The type and data elements within your patient administration system will affect the level of technological change and your ability to automate survey administration and analysis
• Existing familiarity across the organisation with the purpose and benefits of patient experience measurement will affect the level of initial ‘marketing’ required
• Existing familiarity with integrating patient perspectives and safety and quality data into the quality improvement cycle will affect the level of process change and training required
• The financial and human resources available for the implementation will affect your ability to conduct pilots and analyse these thoroughly (but not conducting a pilot may be a false economy if problems emerge)
• Level of in-house capacity and available skills will affect whether contractors need to be engaged.

Staged implementation and piloting

Deciding how to take the PREM into your organisation is a strategic decision and could have important implications for stakeholder engagement and for how the results are used. Rather than implementation across your whole organisation or patient population all at once, you might consider a staged implementation which includes pilot sites. The advantages of this are that:

• Lessons from pilot projects can be applied to improve the implementation process across other settings
• Value can be demonstrated to sceptics early on, to increase overall engagement 
• Overall costs of implementation may be less because the need to correct problems on a large scale is lessened if they are identified during pilots
• The pilot surveys can be used to ask patients what they thought of the survey – such as its length, relevance and layout – and to make appropriate adjustments.

Prioritisation of pilots

If you decide on a staged rollout, you will need to consider how you prioritise your pilot samples and analyse your pilot data. You will need to develop a rationale for your staging and choice of pilot populations or sites. Some ways to prioritise samples for piloting include:

• By ward or department (this may be a good choice if you already have clinical buy-in)
• By reason for admission 
• By type of admission (day stay or overnight).

Evaluation of pilots

Consider how you will evaluate the pilots and feed these findings into the full implementation. You could consider:

• The implementation process itself
• Stakeholder engagement
• Survey response and completion rates.

Parallel surveying

If you are planning to replace your existing PREM or add to it, you might consider a parallel survey process. This is where you continue to administer your old survey to some patients and at the same time administer the new survey to other patients. The advantages of this are that:

• It gives you more time for pilot testing without a break in patient surveying
• Comparative analysis of the results of the two surveys can yield very useful data to guide your implementation and presentation of results, including
  • how response rates and completion rates compare – if they are higher for the updated or new PREM, this helps prove value; if lower, this can inform adjustments to the mode of administration or format of the survey
  • how overall scores compare between the two surveys – this will help to set appropriate levels of expectation for scores on the new survey.

Outcome

By completing step 2.2 you will have a ‘sampling frame’ for your full implementation. This is separate to any sampling frame for pilots. It is the population of patients you would like to include in your regular surveys when you have fully implemented the PREM. This means deciding what types of patients will be eligible for the survey, what proportion of these patients you will invite to answer the survey, and the minimum number of completed responses you will require before you can draw valid conclusions from (or report) the data.

Things to consider

This section lists the items that need to be considered in Step 2.2 to define your sample.

Defining your eligible patient population

• What types of patients would you like to send the PREM to? To ensure your PREM results represent your patients and provide a good overview of your services, you should consider the different service areas within your organisation, and the demographics of your patients.
• Are you interested in all patients, or only a subset? If you are interested in a subset, what are the attributes of the patients you want to be eligible to complete the survey? Depending on your objectives for using the PREM, you could restrict your sample in various ways (or a combination of these), for example:
  • by age
  • by condition
  • by hospital, clinic, department or ward
  • by type of service received
  • by length of stay 
• How will you automate the process of identifying your survey sample? Does your existing patient administration system allow you to filter patients by the attributes you are interested in?

Exclusions

You may decide to exclude some patients from your survey population on ethical or pragmatic grounds. These exclusions will depend on your organisation and the type of care it is responsible for, but you might need to consider how to treat patients who:

• Are having treatment which means they have repeated visits within a short time period (for example, chemotherapy)
• Are likely to be surveyed using the same or a similar survey by two entities (for example, mental health patients who are given the Your Experience of Service [YES] questionnaire; patients of privately owned public hospital services where a state government and private hospital group might both ask the questions)
• Are mothers who have experienced a stillbirth
• Are experiencing temporary or permanent loss of mental capacity 
• Visit the emergency department but are not admitted to the hospital 
• Have diagnosis codes and types of health service which were excluded from those excluded from the PREM development process already tested (only if you want to be able to claim validity and reliability without further testing).

Sampling from your eligible patient population

Whether you want to know about the experiences of patients across all demographics and all services or only a subset of these, you still have to decide what proportion of the eligible patient group you would like to survey. The decision will be affected by your chosen mode of administration (which determines the cost of each completed response).

You can either ask for responses from every discharged patient in your chosen population or you can develop a sampling frame to determine how a subset of those patients can be chosen.

Sample stratification

If you are not intending to send the survey to all eligible patients (and have decided to take a sample) you now need to decide whether to stratify the sample. This means dividing your eligible patient population into mutually exclusive groups based on a variable of interest (for example, age, admission type, department of admission) and then sampling from each of these groups. This reduces the problem of random sampling from a population where you may entirely miss respondents with some attribute of the variable you are interested in.

Sample size calculation

If you are not intending to survey all eligible patients, you may wish to draw conclusions from your sample and assert that these apply to your whole eligible population within a margin of error (using confidence intervals). To do this, you will need to set a minimum number of completed responses to include in your analysis, to achieve statistical power.

For PREMs aimed at local quality improvement, the results can be used at local level to improve the service's responsiveness to patients' views, to implement feedback loops to ensure patients receive the appropriate response to any issue raised, and to point to issues for further investigation.

Outcome

By completing step 2.3 you will have a plan for how often you will administer surveys to your sample or population of patients, and when you will administer the survey. These timing decisions will need to support your objectives for using the PREM. For example, if you plan to report to your board or other regular meetings, your reporting cycle will affect your surveying and analysis cycle.

Things to consider

This section lists the items that need to be considered in Step 2.3 to decide on the frequency and timing of the survey in a hospital setting in relation to a person’s discharge. For non-hospital health settings, apply aspects of these considerations as relevant and as it relates to the completion of a person’s episode of care. 

Whatever timing you choose for your survey administration, the key is applying this consistently.

Options for timing of administration

There are several options for the timing of the survey:

• Option 1 – rolling (continuous) administration triggered by discharge (captures all eligible discharges); administration date is determined by elapsed time since the individual’s discharge
• Option 2 – periodic administration at regular intervals (captures all eligible discharges since last administration); administration date is determined by date of previous administration
• Option 3 – periodic cross-sectional administration (captures eligible discharges only from a certain time period rather than all eligible discharges).

Options for timing in relation to each patient’s discharge

There are several options for the timing of the survey in relation to patient discharge:

• Option 1 – immediately before discharge
  
  The advantages of this option are that you can use consumer liaison workers or volunteers to administer the survey in person using a tablet device (Computer Assisted Personal Interviewing, or CAPI), which may increase response rates. This option has to be done while the person is waiting to leave, to prevent any fear of it affecting their care. The disadvantage of this option is that the discharge experience will not be reflected in responses, it is labour intensive, and there may be a risk of interviewer effects and social desirability bias in the results.
• Option 2 – 24–48 hours after discharge
  
  The advantages of this option are that the patient’s experience is fresh so it may increase response rates; recall bias may be reduced and there is less risk of the patient confusing this experience with a different one. The disadvantage of this option is that the person may still be too ill to feel like responding. For most types of health services, early surveys tend to get more negative responses than more delayed surveys.
• Option 3 – within a week, fortnight or month of discharge
  
  The advantages of this option are that the patient will likely be less affected by physical pain or discomfort and may have a more realistic reflection on their experience; reflections on discharge and follow-up can also be captured. The disadvantage of this option is that the patient is more likely to confuse this experience with others, especially if asked more than a fortnight after discharge; this may increase recall bias.

Outcome

By completing step 2.4 you will have decided whether and why you would like to adapt some aspect of the selected PREM, and you will understand the implications of doing this.

Things to consider

This section lists the items that need to be considered in Step 2.4 to decide whether you need to adapt some aspects of the PREM.

Reasons for adapting the PREM

You may wish to adapt the PREM to better suit your own circumstances. Reasons for adapting the PREM may include:

• Using the PREM in different types of services to those in which it has been tested
• Using the PREM in different types of patients to those with whom the PREM was tested
• Ensuring the concepts addressed in the questions are culturally appropriate for diverse populations
• Keeping some questions from an existing survey to preserve time series data or to reflect organisational priorities
• Adding supplementary questions around the PREM to target issues of local relevance or to get more detail (for example, qualitative content) or context.

Reliability and validity of the PREM

A PREM cannot be considered valid or reliable unless you use the questions in one of the ways they were originally tested. If you want to establish reliability and validity for other modes of administration, ‘nesting’ the questions within other surveys, adapting any question, reordering or interspersing questions, or use with other patient populations or service settings, you will need to do your own field testing and statistical analysis.

Bear in mind that if you adapt a PREM in any way you may need to adhere to license requirements, which may include attributing the original questions to the PREM developer and noting how you have adapted them.

Types of adaptation

Examples of ways you might consider adapting a PREM include:

• Including the PREM as a module in a bigger survey
  You may wish to ‘nest’ the PREM as a module within a larger set of patient experience questions. If you are doing this, it is preferable to keep the PREM questions together in the same order, retain the rating scales for each question, and use all of the questions. Some implementers add net promoter scores, an opportunity for free text comment, and questions which are particularly relevant for their organisational priorities or to continue a time series from an old survey.
• Using the PREM plus local options
  Within a group of hospitals or services, the same core set of questions of the selected PREM can be asked across the whole group, with freedom for each hospital or service to add in additional questions that suit their particular local circumstances, quality improvement initiatives or strategic objectives. 
• Adapting wording or content
  Adapting the PREM wording or response options is generally not recommended unless you are going to do a field test and statistical analysis of the new wording for a different patient population. 

Any of the above may be temporary adjustments to meet particular organisational priorities or support quality improvement initiatives.

Free text questions

Incorporating an opportunity for the patient to give a free text comment may be valuable. This is usually in the form of a general question about what made the respondent rate their overall experience in the way they did. Advantages of adding such an opportunity are that:

• The reasons behind a person’s multiple choice PREM responses become easier to interpret and act on
• Common themes from free text responses can help early detection of emerging patterns of good or poor practice, can be used in training, and can be used to feed compliments back to particular staff 
• Any remedial action required by the health service can be done on a case-by-case basis in a timely way (if the person discloses an incident, near miss or other concern about safety and quality)
• Free text responses are valuable ‘safety valves’ for patients who feel that the PREM questions did not ‘get at’ their main issues or feedback.

Outcome

By completing step 2.5 you will have decided the mode of administration of the survey and how you will present it to patients, as well as what you might need to do to accommodate the needs of different patient groups.

Things to consider

This section lists the items that need to be considered in Step 2.5 to administer the PREM to patients.

Formats – options and influences on choice of option

There are three main formats that a PREM may be tested on: online, pen and paper, and computer-assisted telephone interview (CATI). In deciding which of these to use, you will need to consider:

• Patient demographics (for example, are your patients mainly older people who may find pen and paper more useable than online)
• Resource capacity (for example, do you have the IT resources to conduct online surveys, do you have staff available for data entry of pen and paper responses).

You may find that you will need to implement a combination of formats, particularly if you have a wide range of patients – some may prefer online, some may prefer pen and paper. Also, even if you implement an online or CATI survey, you may need to provide other options as a back-up.

You also need to consider whether you will need external resources to conduct (or help conduct) the survey. External companies can also be contracted to conduct the entire process from survey to reporting, or can just provide one part of the process.

Meeting patient needs

You will need to decide how to capture the experiences of patients with special needs. Consider (and if necessary test) appropriate modes and/or formats for the survey for:

• Patients with sensory or cognitive impairment (for example, CATI, proxy respondent)
• Patients who cannot read (for example, pictorial or audio version, CATI)
• Non-English speakers
• Culturally diverse populations who may understand health, illness and health services in a different way.

Communicating with patients

If you are to receive adequate numbers of responses to the PREM, you need to ensure that patients are engaged with the process. Communication is the key.

Start early during the patient journey. For example, the patient can be first told about the survey during an episode of care, so that approaches once they have completed their treatment do not come as a surprise. 

Make sure the materials you provide focus on the survey as a way to improve the quality of care. Ensure they are clear, attractive and written in plain English. You will need to consider:

• In-building promotion such as posters
• Cover letter or email
• A patient-facing title for the survey (making it clear what the survey is for) 
• Brief introductory text within the survey 
• Instructions for survey completion.

Outcome

By completing step 2.6 you will have a set of principles and actions to ensure you protect patients’ privacy and use their information ethically when collecting, analysing and reporting PREM results.

Things to consider

This section lists the items that need to be considered in Step 2.6 to ensure ethics and privacy requirements are followed.

Your methods of ensuring patients’ privacy and treating them and their information in an ethical manner may differ according to local requirements and operational processes. 'Ethical' conduct of the survey and treatment of data relies on good governance and is separate to formal ethics approvals processes which are required if you intend to publish in peer-reviewed journals. This Step examines aspects of ethical surveying whether or not you go through formal ethics application processes.

Initial approach to patients

It is good practice to give patients advance notice that they will later be given a survey about their experience. There are several options for this initial approach to patients, including:

• Giving written information on admission to enable informed consent to participate 
• Advising as part of discharge paperwork that a survey will be coming
• Personal visit from a consumer liaison worker or volunteer to explain the survey 
• Letter or email after discharge to forewarn of the survey’s arrival.

As part of this advance notice, and in order for consent to be informed, it is important that it is made clear to patients that:

• They can ask any questions they have before completing the survey
• Their responses are or are not anonymous; if not anonymous, who can access their information
• It is voluntary for them to participate in the survey
• Their responses will in no way influence the care or treatment they will receive in future
• They may be contacted to follow up on their responses, and in what circumstances that may happen
• The information they provide will contribute to improving quality, safety and other patients’ experiences
• Their information will be kept confidential, stored securely, and aggregated and de-identified for analysis.

Consent to participate

Patients should be given the opportunity to provide consent to participate in a PREM program. This consent can be either explicit or implied:

• To give explicit consent, the patient must be given materials that explain the PREM and what is involved for the patient, or discuss the PREM with a staff member with a checklist of information, and sign a document expressing their understanding and willingness to participate
• To give implicit consent, the patient must still be provided with materials or talk about the PREM with a staff member; their response to the survey is then taken as ‘implied’ consent that they have agreed to participate.

Follow-up procedures

You will need to consider how you will handle follow-up, especially where patients are interested in further dialogue with the hospital. Consider giving an opportunity within the survey for anyone who discloses harmful or unsafe practices to ask to be contacted (or to contact someone at the service). Also consider whether and how you will create a feedback loop to let the patient know how their feedback has resulted in change or to thank them for a compliment. This is easier when there is a free text question.

Outcome

By completing step 2.7 you will be able to develop a set of practical tasks which need to be completed to facilitate the processes of getting the survey to a patient and getting it back. This task list will help to determine the initial and ongoing costs and human resources required for administering the survey.

Things to consider

This section lists the items that need to be considered in Step 2.7 to develop a task list to facilitate the survey process.

Sampling logistics

Existing patient administration systems can help you automate identification of your eligible sample and reduce the amount of demographic information you need to ask for in the survey itself. If information is linked to survey responses in this way, this needs to be explicitly described in the information materials that patients are given before they agree to participate in the survey.

De-identification can still be achieved, or identification restricted to a small number of people in particular circumstances, by substituting a patient identifier for name and address in the patient experience data collection.

Surveying logistics

Consider current patient experience surveying methods and whether the new question set can be ‘slotted in’ to existing software or processes without any other changes. Depending on the mode of administration, bespoke surveying software, phone interview resources, or printing and mailing services will need to be arranged in-house or contracted.

Consider who will be responsible for each stage of the surveying process. For example:

• Who or what will trigger the sending out of a survey?
• Where will the responses ‘land’ when they are returned (which email account or database or physical location)?
• Who will clean/analyse/report the returned data?
• To what extent will the sending out, analysis and reporting of responses be done in a central or distributed way? This is especially relevant if your organisation is responsible for a number of services or a geographic area.

You may also need to consider strategies to improve response rates (for example, multiple formats, reminders, patient self-registration portal).

Reporting logistics

This links strongly to your objectives for using the selected PREM. To achieve these objectives, you will need to consider how you will produce reports, how often, what form they will be in, and who will be the audience.

This also links to the mechanisms that you will use to achieve your objectives. Consider what automated processes will need to be in place to ensure results are regularly integrated into other processes and initiatives in the organisation (for example, integration with complaints management, incident management, quality improvement systems and accreditation).

• Greenhalgh T, Robert G, MacFarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. The Milbank Quarterly 2004;82(4):581–629.
• Institute for Healthcare Improvement (US). How to improve. Boston, MA: Institute for Healthcare Improvement; 2018 [cited 2018 Jun 26].
• Nilsen P. Making sense of implementation theories, models and frameworks. Implementation Science 2015;10:53.
• Queensland Health. Queensland Stay On Your Feet® Community Good Practice Toolkit – Phase 1 materials: conducting surveys/questionnaires. Brisbane: Health Promotion Unit, Division of the Chief Health Officer, Queensland Health; 2007.
• Statistics Canada. Survey methods and practices. Ottawa: Statistics Canada; 2010.
• Weiner BJ. A theory of organizational readiness for change. Implementation Science 2009;4:67.