Implementation basics

AHPEQS is available free of charge for use by hospitals and healthcare services to survey patients, and can be adapted for local use if licence conditions are met.

Print

AHPEQS implementation

The Australian Commission on Safety and Quality in Health Care has developed a two-phase approach to AHPEQS adoption in Australia:

  • Phase 1 – local implementation and learning from early adopters

    AHPEQS has been released for use by hospitals and healthcare services. Early adopters are already using AHPEQS within their own organisations. We are providing resources and drawing on the experiences of these early adopters to encourage and support the use of AHPEQS more widely, including establishing a community of practice for implementers to learn from each other.

  • Phase 2 – national specification

    We plan to use the evaluation of phase 1 to build detailed technical specifications to support the establishment of AHPEQS as a national standard tool to assess patients’ experiences all over Australia. The second phase will depend on extensive negotiations and agreement by all sector and jurisdictional stakeholders. (See also Future requirements.)

How AHPEQS is used in health services

AHPEQS is designed so that it can be implemented at any level of health service delivery:

  • Across a whole state or territory
  • Across a whole sector
  • Across a whole private health organisation
  • Across a hospital or day-stay clinic
  • Across a group of hospitals or day-stay clinics
  • In individual services, wards or departments.

AHPEQS can be used alone or with existing patient experience surveys, as a short additional module.

How AHPEQS can be given to patients

The Commission tested three survey types to enable organisations to administer AHPEQS in the way that best suits their needs and those of their patients. These are:

  • Online
  • Pen and paper (PAPI)
  • Computer-assisted telephone interviewing (CATI).

Early adopters of AHPEQS have been trialling other modes of administration including:

  • Text message
  • Computer-assisted personal interviewing (CAPI) on tablet devices.

AHPEQS is designed with different modes of delivery to suit many healthcare environments and existing survey programs. Combinations of modes can be used depending on the circumstances. For example:

  • A state government may prefer to use AHPEQS using CATI
  • A hospital may decide to use a mix of online and PAPI surveys
  • A ward may find that a PAPI survey works best for their type of consumer.

Note that AHPEQS was tested in CATI, PAPI and online forms. Questions which performed very differently in tests of reliability and validity between these three formats were discarded. However, direct comparison between responses received using different modes of administration is not advised.  More information is available on request. 

Online administration

The easiest method of administering AHPEQS to patients is by sending a link (by text message or email) to an online survey form. There is a variety of free and subscription-based online survey tools available, and you can set up AHPEQS on one of these. Many of these work on mobile devices for patients’ convenience. There are also companies that can provide a full and automated survey administration, analysis and reporting service.

Issues to consider when choosing online administration of AHPEQS include:

  • Where the information that patients submit will be stored (some online survey providers are hosted overseas); your organisation should comply with relevant Australian privacy legislation
  • Availability of patients’ email addresses or mobile phone numbers to send the survey; if your organisation does not routinely collect email addresses, an alternative could be to post online survey links, or send a link in a text message to a mobile phone
  • Potential biasing of the sample if population coverage is poor (for example where substantial numbers of patients do not have email addresses or access to the internet); an alternative survey option may be needed to account for this (for example, pen and paper format).

Health service organisations may also wish to consider the use of computer-assisted personal interviewing (CAPI), where an interviewer talks to the patient in person and uses a computer (usually a tablet) to answer the questions.

Pen and paper

Printed copies of AHPEQS can be sent by post to patients, together with reply-paid envelopes to return the survey.

Issues to consider when choosing pen and paper administration of AHPEQS include:

  • The need for accompanying documents to the survey itself, such as an information sheet and reply-paid envelope
  • The resources required to print surveys, pay for postage and manually process or scan responses 
  • Likely higher rate of skipped or ambiguously answered questions.

Computer-assisted telephone interviewing (CATI)

CATI is a surveying technique where the interviewer asks questions over the phone, following a script provided by a software application. CATI should be conducted by professionally trained interviewers and requires special software to guide the interview process and to enable the interviewer to record the answers. It is expensive to conduct, but completion and response rates tend to be better than other methods of surveying.

Issues to consider when choosing CATI administration of AHPEQS include:

  • The requirement for interviews to always be conducted in exactly the same way (this will involve quality monitoring for at least 10% of the interviews)
  • The requirement for interviewers’ judgement in applying a ‘silent’ response category if the respondent cannot answer a question
  • The potential requirement for a large number of attempts to talk to the relevant person.

Making changes

Individual organisations can use and adjust AHPEQS to meet their own needs. However, you should be aware of potential problems with validity and reliability, and future requirements.

If you adapt the AHPEQS questions or how AHPEQS is administered, or use it in other populations, its validity and reliability cannot be assured unless further statistical testing of collected data is carried out. This does not mean that the tool cannot still be useful for local monitoring and informing quality improvement, as long as it is administered and analysed in a consistent way within your organisation.

Validity and reliability

Field testing of a draft version of AHPEQS was conducted with a total of 1,460 recent patients across four Australian states. The purpose of the field testing was to establish the reliability and validity of questions, and to remove any questions which did not perform well in statistical testing.

The field testing of the draft AHPEQS questions incorporated three modes of administration: computer-assisted telephone interviewing, pen and paper by post, and online self-administration. Patients were surveyed two months after discharge, and a retest survey was administered two weeks after that. 

The following tests were performed for validity and reliability:

  • Validity – content validity; construct validity (factor analysis); convergent validity, using an additional question (Would you recommend this hospital to family or friends?)
  • Reliability (test–retest) – Cohen’s kappa; PBA kappa (adjusting for skewed response distributions).

As a result of field testing, AHPEQS has so far been validated and found to be reliable in certain types of healthcare setting using certain types of survey delivery. Those included in the field testing were patients who:

  • Were aged 18 or over at the time of completing the survey
  • Had spent at least one night admitted to a public or private hospital OR have been admitted to a private day-only clinic for a day procedure
  • Had been discharged from this overnight or day stay in the calendar month two months prior to survey administration 
  • Were not primarily admitted for mental health reasons, maternity services, chemotherapy or renal dialysis (a full list of sample exclusions is available on request).

The three modes of administration for which validity and reliability was established in these populations were computer-assisted telephone interviewing, online self-administration, and pen and paper by post.

Organisations are free to use AHPEQS in settings and with types of patients which were not part of the original field testing, but will have to conduct their own pilot process to establish validity and reliability in those other circumstances. Early implementers are testing AHPEQS in a variety of settings including outpatients, emergency departments, paediatrics and community services. They are also trialling other modes of administration including bedside computer-assisted personal interviews with tablet devices.

Detailed information about the development and testing process and sample exclusions will be available in forthcoming technical reports. A summary of the process is provided below.

AHPEQS Development and Testing - Summary (767.5 KB)

Future requirements

The Commission plans to develop national technical specifications for AHPEQS in 2019. It may be decided at a later date (in consultation with governments and private health providers) that AHPEQS will be used for nationally consistent measurement, reporting or benchmarking. If this occurs, organisations will need to implement AHPEQS in a standard way, and the Commission will provide detailed guidance about the format and methods to be used as well as national reference datasets and toolkits.

Licensing and copyright

Copyright for AHPEQS is owned by the Australian Commission on Safety and Quality in Health Care. AHPEQS is available free of charge under a Creative Commons Attribution-Noncommercial-Share Alike (BY-NC-SA) v4.0 licence. This licence gives users the right to copy, adapt, modify and distribute the work (as long as they adhere to attribution requirements shown in the document linked below), and to licence it to others on the same terms for non-commercial uses only. If you use or adapt AHPEQS, you must adhere to copyright and licence requirements set out in the following document.

AHPEQS licensing and copyright requirements (145.55 KB)

Using the Stage 1, Stage 2 and Stage 3 guidance

The Commission has developed information to guide you through the three stages of implementing AHPEQS and using the resulting data:

Relevance to organisations

The information in this guidance is relevant to any type of organisation or department that funds, manages or provides hospital or day surgery services to patients, from government to local unit level. The information is equally valuable to organisations where the decision to use AHPEQS has already been made and to organisations where the use of AHPEQS is still being considered. Even if you have been told by your head office or government that you have to use AHPEQS, it is important that you ensure that the results are used in the most beneficial way for your particular organisation.

Relevance to staff

The information in this guidance is useful for anyone who is responsible for ensuring the voice of patients and consumers is routinely listened to and used in their organisation’s decision-making. This could include anyone working in a health service organisation, from executive staff members to healthcare staff, consumer engagement professionals, or quality and safety professionals.

Different groups may take the lead at different stages. For example, executives may wish to think through the considerations in Stage 1, before handing Stage 2 on to safety and quality managers. Consumers should be actively involved in decisions at every stage of implementation and operation of AHPEQS.

Recommendations

Throughout the three stages, we present suggestions for things you might consider to ensure that your implementation of AHPEQS is well thought through and useful. There is no implied endorsement of particular suggestions. They are provided as prompts to help you to think through how you can use AHPEQS to meet your organisation’s particular priorities and needs.

AHPEQS implementation resources and references

Back to top