Use of an audience response system to maximise response rates and expedite a modified Delphi process for consensus on occupational health

The Delphi method was originally developed by the US Rand Corporation as a method for decision making and forecasting [1–3]. The technique used an iterative approach for obtaining consensus. It appeared to be especially useful in the absence or non-availability of definitive data, and when expert opinion is therefore sought as a possible alternative to decisions based on published evidence. The subject area in which it was first used was defence, but the technique was then adopted by other disciplines, such as business, social science and medicine for research, education and training purposes [4–10]. The Delphi approach allowed agreement to be reached, even in areas as potentially contentious as determination of priorities, provision of services and barriers to delivery. The technique has also been used for occupational health in several countries, including Malaysia, the United Arab Emirates and the United Kingdom [11–15].

One of the limitations of the multi-stage iterative process is the low response rate, especially during the subsequent rounds of iteration, as it has been experienced in various settings [16, 17]. This has led to considerations as to whether there is a possibility of completing the exercise in a shorter period of time to reduce poor participation from the first round through to completion. Single-round, almost real-time methods and use of online systems have been proposed [18, 19], and researchers have attempted to modify the method to improve participation rates and shorten the process time. We have trialled the use of a computerised audience-response system for determining key issues and barriers to effective provision of occupational health in the United Arab Emirates (UAE) [20]. The method was refined and used in 2014 to determine consensus on health and safety issues and obstacles to successful implementation of health and safety programmes for a large utilities company in Abu Dhabi.

On the day of the exercise, the stakeholders were instructed on the necessity to provide their own views without involving any discussion with their colleagues. The importance of obtaining a personal individual opinion was stressed. Each participant was provided with a ‘clicker’ (Keepad Interactive, NSW, Australia; see illustration, Fig. 1) and received oral instructions and practice sessions on how to use the device.

The study was approved by the Social Sciences Research Ethics Committee of the United Arab Emirates University.

The two exercises in Abu Dhabi showed that the modified Delphi method using an automated audience response system had many advantages in obtaining a consensus view from stakeholders on pre-determined issues in occupational health, most importantly the exercise could be completed in a short period of time with a very high response rate in the consecutive rounds. The use of a combination of written responses and the computerised audience response system proved effective in enabling a multi-round iterative exercise to be conducted over a day, thereby reducing dropouts from participation. The majority of previous Delphi projects have taken several weeks or months [5, 14]. This was because several iterations were required, and a postal or online system was used to gather information, with time allocated between iterations in order for responses to be received.

The computer-aided audience response system also enabled almost immediate feedback to the group about their responses at specific stages of the modified Delphi study. Stages 2 and 3 of the iterative process could then proceed after participants had received feedback from the preceding stages, allowing them for comments and questions. At the end, the final results of the exercise were shared with the participants so that everybody could express their final opinion. Informal feedback through initial provision of the findings to the company suggests that there was general agreement with the priorities and barriers identified through the Delphi exercise.

Completion of all stages in a day reduced the dropout rate. The final participation rate for multi-stage Delphi exercise is often low and would be affected by the number of rounds involved, and the time given between stages. A recent study employed a two-round online Delphi study to develop consensus on remote healthcare practitioners’ competency for oil and gas operations and the valid response rate for round one and two was 27 and 24 %, respectively [17]. Study authors reported that technical issues with the recruitment, sampling and data collection techniques accounted for 59 % of the non-response across the two rounds: out-of-office replies (26 %), incorrect email addresses (19 %), and burst rate limit exceeded (14 %) [17]. In the two-stage postal exercise on priorities for occupational health research involving 53 senior occupational physicians, Harrington obtained an 86 % response to round 1, and 91 % for round 2 [12]. When the exercise was repeated as a postal questionnaire to personnel managers, the response rate was 24 % despite two reminders and a second mailing to non-responders. For our three-stage Delphi project in 2009, the participation rate was 94 %, and in 2014, 83 % of registered participants completed all three stages of the exercise. With a postal questionnaire the participation rate often decreases after each round. Follow up of non-responders is difficult if there are concerns about preservation of the confidentiality of responses. With our modified system, all participants were requested to remain until all stages were completed. Despite the whole process taking only one day, a 17 % drop-out rate was still experienced, with a few individuals leaving before completing either stage three or stages two and three. This may be partly due to inadequate communication or commitment by the participants regarding the importance to remain till the end of the exercise.

Although neither the Delphi process nor the computerised audience response system was familiar for the participants, the majority could understand and complete the expected tasks. The modified approach enabled individual views to be expressed, with equal weighting given to each person’s opinion, regardless of position or rank within the organisation, or professional or managerial background or experience. The likelihood of opinions being influenced by the views of senior or more vocal colleagues was reduced by providing an individual ‘clicker’ for each participant. Individual participant choices could then be transmitted to the computer with preservation of anonymity. Participants were reminded to provide their own opinions without group discussion, and that the projects were designed to obtain individual opinion and not necessarily company policy or views of the group. Therefore, there was a strong likelihood that the views of individuals were obtained. With postal or even online questionnaires there is no certainty as to whether the views expressed or the choices made were completed as a group exercise or completed by persons other than the identified respondent.

A major limitation of the modified approach is the requirement for stakeholders to be physically present to participate for a day. This requires time away from work, and the costs of accommodating a large number of individuals including finding a suitable venue and providing meals and refreshments. This was feasible in the two projects in Abu Dhabi as there was commitment and support from the employers for the process. There is also a need for a small team of experienced support staff to facilitate the distribution and collection of questionnaires and ‘clickers’, and to analyse data captured by the audience response system in a limited time between the stages. The availability of a set of 100 clickers (to include spare devices) that had to be checked and tested ahead of the exercise was essential. The process required a smooth transition from one stage of the Delphi procedure to the next, without unnecessary delay that could well reduce the likelihood of participants remaining for subsequent stages. Good audio-visual facilities were essential, and a preliminary explanation as to the purpose of the process, and the importance of full participation was key to the success of the modified Delphi study.

The selection of participants was not random therefore it did not assure that opinions were representative for the views of the organisations they represented. Nevertheless, random selection is not the main purpose of selecting participants for a Delphi exercise; rather, purposeful sampling was used to recruit knowledgeable and experienced participants on the topic of interest [10]. This concept was obtained by inviting acknowledged professionals from the field of occupational health and safety available in the United Arab Emirates in the 2009 study. In 2014, the organisers of the initial survey within the company were entrusted to recruit employees related to occupational health and safety issues from various sectors of the company in a number that is manageable in a one-day exercise. The choice of participants with different levels of seniority, representing different professional groups or working in different geographical areas allowed views to be gathered from a wide area of representation.

The presented modified Delphi technique with the use of a wireless audience participation system enables rapid consensus that proves especially advantageous in effectively identifying priorities and barriers of occupational health and safety in countries and workplace settings where such knowledge is particularly needed.