Step 3.2 – Design an analysis strategy
Outcome: By completing Step 3.2, you will have an analysis strategy for the raw AHPEQS data you will receive from patients.
Things to consider
This page lists the items that need to be considered in Step 3.2 to prepare an analysis strategy for the raw AHPEQS data.
Processing and ‘cleaning’ raw data
Depending on your mode(s) of administration, different methods for processing the raw data received from patients will be required. The goal of this processing is to deal with any ambiguities in responses and get the data into a format that can then be read by analysis software.
For example, if automated scanning technology is not being used, paper surveys may require manual data entry into a database and a rule will need to be developed for interpreting unreadable, ambiguous or incomplete responses and reflecting these in the database. A rule would also need to be developed for dealing with anything written by the respondent that does not fit with the response options given (for example, a request to be contacted if that possibility has not been offered).
The cleaning process also can involve de-identification of data – such as replacing a person’s name with a unique identifier or separating their identifying details from their survey responses.
Descriptive statistics
Once the data have been ‘cleaned’, descriptive statistics can be used to summarise and describe the basic characteristics of aggregated survey response data for a cohort of patients. This is the simplest analysis strategy and can be represented in tables, charts and line graphs. Examples of descriptive statistics for a set of AHPEQS responses might be:
- Frequency of response option choice for each question
- Trends over time in frequency of response option choice for each question.
Scoring methods
Along with simple frequency analysis, you may wish to ‘score’ responses. Scoring (coding) is a way to convert qualitative responses (such as multiple-choice survey responses) into numerical data. Applying a score means placing a numerical value on response options in a way that reflects the desirability of that response.
You do not have to apply a scoring method, but, if you do, it must be applied consistently across all responses received from patients, whatever the mode of administration. No matter what the type of response options (for example, yes/no, ‘always’ to ‘never’), the most desirable response for each question needs to be assigned the highest or lowest score in a consistent way.
Pilot work showed that many healthcare services score highly on most AHPEQS questions. To discriminate between excellent and good experiences, and to motivate improvements towards consistent excellence, it can be useful to only count ‘top box’ responses. Results are then presented in terms of the proportion of all responses for that question which received a top box response (for example, ‘always’). Other responses are not broken down in reporting. Anecdotally, this approach can be more effective in catalysing quality improvement and behaviour change within an organisation than the partial credit system.
If you are using a ‘top box’ system, it can be useful to add analysis of ‘bottom box’ (that is, the least desirable) responses as well. This can help highlight problematic patterns in quality and safety that would require corrective action.
Partial credit scoring
In AHPEQS, most of the response options are offered on a frequency scale (‘always’ to ‘never’). For most of these questions, ‘always’ is the most desirable response. Using a ‘partial credit’ system, the highest score would be applied to that response, and progressively lower scores to ‘mostly’, ‘sometimes’ etc. Some advantages of applying partial credit scoring are that:
Some disadvantages of applying scoring and aggregation of scores are that:
- if you are required to report a composite score for performance on the whole AHPEQS, scoring can help with this process.
- when the scores on each question are aggregated across a group of patients, an average score can be calculated
- credit is given proportionately for each option, so that services which get all ‘mostly’ responses will ‘perform’ better in their overall scoring than those services which get all ‘rarely’ responses
- assigning numbers to qualitative categories (such as ‘always’) can lead to misleading representation of data (for example, giving a score of 4 to ‘always’ may imply that this is twice as valuable a situation to the patient as ‘sometimes’ which gets a score of 2)
- a decision needs to be made about how to treat different types of response option (for example, yes/no vs frequency scale)
- a decision needs to be made about how to treat missing or ambiguous responses.
‘Top box’ and ‘bottom box’ scoring
Pilot work showed that many healthcare services score highly on most AHPEQS questions. To discriminate between excellent and good experiences, and to motivate improvements towards consistent excellence, it can be useful to only count ‘top box’ responses. Results are then presented in terms of the proportion of all responses for that question which received a top box response (for example, ‘always’). Other responses are not broken down in reporting. Anecdotally, this approach can be more effective in catalysing quality improvement and behaviour change within an organisation than the partial credit system.
If you are using a ‘top box’ system, it can be useful to add analysis of ‘bottom box’ (that is, the least desirable) responses as well. This can help highlight problematic patterns in quality and safety that would require corrective action.
Analysis infrastructure
Think about how you will automate the analysis. This will involve some kind of database to store the raw data, with software to enable simple manual data entry and/or to enable automatic feeding in of returned electronic survey responses.
Security of stored data
Think about:
- How you will store data
- How long you will store data
- Who will have access to the data, and your arrangements for preventing access by other people
- Whether all stored data will be de-identified
- How you will communicate your security arrangements to consumers, to reassure them about the survey processes.