ABSTRACT
In the last 30 years, the application of the Delphi technique has been increasing. With the recent availability and established popularity of Internet-based research tools, the Internet has been identified as a means for mitigating Delphi limitations, maximizing its advantages, and expanding the breadth of its application. The discourse on the application of 'e-Delphi' has been speculative in nature, however, with pragmatic analysis of Internet-based Delphi administration being limited in scope. Concomitantly, methodological guidance for conducting the e-Delphi in health education is limited, with best-practice implementation strategies yet to emerge. This paper advances the understanding of the e-Delphi technique, contributes to the evolution of this Internet-based research methodology, and provides guidance for the e-Delphi researcher in health education. Further, it offers a contribution to the discourse about Internet- based inquiry generally, and on using the e-Delphi technique in health education specifically. The authors illustrate that e-Delphi has a range of benefits that are effective and efficient in assuaging traditional Delphi limitations; nevertheless, a set of methodological issues remain unaddressed and make apparent the need for future research investments to better understand and alleviate challenges presented for e-Delphi research in health education.
INTRODUCTION
Since its early conception as a 'galactic network' of globally connected computers, the Internet has moved beyond the realm of science fiction to become the primary global communication medium. Until now, no other communication device has been so readily adopted with such widespread socio-cultural and economic implications. While the complexity of the Internet makes an accurate measure of its size challenging, over 13 billion indexed webpages can be identified using common search engines such as Yahoo!, Google, Bing and Ask. (1) Nearly four out of five people around the world now call internet access a basic human right. (2) Health information remains one of the most important topics that Internet users research online. Eight out of ten Internet users look online for health information, making it the third most popular web endeavour (following email and accessing search engines). (3) Furthermore, wireless Internet users are more heavily engaged than desktop Internet users. Eighty-three percent of wireless Internet users have looked online for health information, compared to 70% of Internet users who do not use a mobile connection. (3) The structure and function of the Internet has revolutionized the way that we access, communicate and share health information.
The Internet revolution has also impacted the manner in which health data is collected, transmitted, stored and analysed. By way of the World Wide Web, and through the interoperability of information systems, the Internet has become an important focus for scientific inquiry in multiple areas within health education and health promotion. For example, Jackson (4) and Leung and Lee (5) explored Internet-based recreation experiences and quality of life. Bargh and McKenna, (6) Engelberg and Sjoberg,; Fogel et al. (8) and Kraut et al. (9) investigated the psychological implications of the Internet as a social venue. Quan-Haase and Wellman (10) and Ellison et al. (11) considered the impact of the Internet on social networking and social support for building social capital. Cassell et al., (12) Eysenbach, (13) Korp, (14) Escoffery et al. (15) and Stellefson et al. (16) examined the efficacy of 'e-health' applications for a variety of individuals and populations. A special issue of the journal Health Education Research focused exclusively on the digital 'revolution' occurring within health education research and practice (see the Bernhardt and Hubley (17) editorial for a synopsis). More recently, in 2010, the American Journal of Health Education put out a 'call for papers' for social media applications in health education.
Whereas the Internet has been a research focus for scholars in health education, the Internet can also be used as a research portal from which data can be accessed. In the past, researchers used the Internet primarily for secondary data collection--that is, data captured from existing Internet-based sources including bibliographic databases (e.g., peer reviewed journals, library collections, etc.), government databases and directories (e.g., policy documents, population census, etc.), maps, audio-visual archives and data archives. (18) New technology-enabled research services have emerged which allow health education researchers to engage in a variety of primary Internet-based data collection activities. However, the academy--in general, has been slow to respond to the Internet's potential as a research portal or medium for data collection purposes). (19,20) In a study by Benfield and Szlemko, (21) it was found that it was not until the last decade that the use of the Internet for primary data collection began to increase. Their search of the Web of Science[R] bibliographic database revealed only 494 peer reviewed articles published between 1996 and 2006 when the keywords 'Internet research,' 'web based research' and 'electronic data collection' were entered.
Nevertheless, recent advances in graphics, software, real-time interactivity and Internet infrastructure continue to reveal new possibilities for experimental studies and both qualitative and quantitative research. (22,23,24) The use of email or an 'e-survey' to collect data are now readily accepted tools facilitating rapid, simple, and inexpensive communication portals for collecting and managing data and participants. (25,26,27,28) Online focus groups, e-conferences, and interviews are bridging geographical, fiscal, and social gaps which previously created chasms between researchers and participants. These former breaches inevitably led to barriers and constraints which presented potentially insurmountable research obstacles. (29,30,31) With the recent emergence of 3G and 4G coverage in smartphone mobile technology, data collection is proliferating, occurring most anywhere in the world, at any time, day or night. Smartphones are used by 74.6 million people in the U.S., (32) and the penetration rate is even higher in Asia and the Pacific regions of the world, including China and India. (33) These statistics will likely do nothing but increase as time and technology progresses.
Thus, studies conducted O'Connor and Madge, (29) Duffy, (23) and Liamputtong (34) suggest that the discourse on e-research needs to be widened to encourage and incubate methodological refinement so that Internet-based methodology can evolve, best practices can emerge, and guidance can be provided for researchers. There is a paucity of practice-based guidelines which exist to direct Internet-based research activities in the health professions. (35) This is of concern to the field, given that enhanced awareness of the value of e-research and its associated challenges is essential to advancing Internet-based inquiry. Therefore, through drawing on antecedent research--recent reviews, select studies and seminal texts, as well as on research observations and experiences, this paper offers a contribution to the discourse about Internet-based inquiry generally, and on using the e-Delphi technique in health education specifically. It draws on the lessons learned from the e-Delphi literature and offers commentary based on concrete research experience. Finally, advantages and limitations of the e-Delphi are suggested, along with recommendations for design and administration in future health education research.
THE DELPHI TECHNIQUE
The Delphi technique is used to systematically combine expert opinion in order to arrive at an informed group consensus on a complex problem. (36) It was originally developed by the Rand Corporation in the 1940s as a tool for soliciting opinion from a group of experts. The Delphi has evolved from its original conceptual and practical applications. Landeta (37) reports that the Delphi is a legitimate and valuable contributor to progress on complex social problems and issues, while de Meyrick (35) reports that the method has a long tradition as a valid research technique--especially in the health science field. Several studies in health education have used the traditional Delphi technique exclusively to determine consensus in a number of important need areas, such as matching manuscript submissions with appropriate health education journals, (38) developing research questions (39) and determining specific topics for inclusion in environmental health curricula. (40)
In principle, the Delphi is a group method that is administered by a researcher or research team that assembles a panel of experts, poses questions, synthesizes feedback and guides the group towards common ground. The Delphi is a method for organizing conflicting values and experiences and facilitates the incorporation of multiple opinions into consensus. (41,42) This is achieved using iterative rounds of sequential surveys interspersed with controlled feedback reports and the interpretation of experts' opinion. Individuals are asked to reconsider their initial positions in light of group trends and can make adjustments to future survey responses accordingly. (43) Survey iteration with feedback is continued until convergence of opinion reaches some point of diminishing returns, which suggests agreement on the topic or problem of interest. Research has shown that the empirical point of stability with regard to consensus for decision making is generally reached after the fourth iteration of responses. (44)
Donohoe and Needham (45) claim that the Delphi's popularity may be attributed to six basic attributes that distinguish it from traditional face-to face group interview techniques. The legitimacy and suitability of the method for highly-complex problems are well-established as are the expert contributions to the understanding and/or the resolution of important problems. The Delphi is particularly well-suited to research problems not compatible to linear or precise analytical techniques, and where subjective judgement on a collective basis could illuminate new perspectives. Anonymity encourages participants to interact free from the constraints of personality conflicts or status relations thereby reducing the risk for group dynamics to negatively influence outcomes. The Delphi is defined by flexibility and reflexivity, as it allows the researcher to adapt the technique to the research context. The researcher can structure the data collection tool--the survey, so as to collect quantitative or qualitative data or both. This allows the researcher considerable flexibility in the design phase, but also enables the collection of a rich and varied data set. Flexibility and reflexivity also allows participants to scrupulously think through the problem between rounds so that the validity of the data and the outcomes are enhanced. Unlike other group methods, the Delphi does not demand the proximity or face-to-face meetings of the researcher with participants. The exercise is conducted remotely thereby reducing travel costs and the potential for group dynamics to manifest.
THE E-DELPHI
By its very nature, the Internet offers a promising and rapidly evolving medium for Delphi research. The conduct of Delphi studies is amenable to the Internet platform where iterative collection of data can be made more efficient. Originally coined by MacEachren et al., (46) the 'e-Delphi' technique represents recent attempts to computerize the Delphi process so as to optimize the method's ability to organize widespread and diverse group thinking, while capitalizing on the aforementioned methodological advantages. The e-Delphi relies on an Internet-based platform for organizing, controlling and facilitating communications between the researcher and expert panel. When compared with the traxditional pen-and-paper approach to data collection, the e-Delphi offers unparalleled convenience, time and cost savings, and data management advantages. Deshpande et al. (47 (p.55)) reported that the e-Delphi is a 'feasible, convenient and acceptable alternative to the traditional paper-based method.' It is important to identify potential research situations where the e-Delphi research technique would be an especially viable alternative, especially as compared to the traditional paper-based method. What follows is a description of one such relevant research situation.
There are important conversations ongoing related to the marketing of the health education profession for the development of the field and the procurement of a sustainable professional base. (48) An important prospective research activity for the field will likely involve the collection of data from prominent experts in health education to facilitate consensus in (a) defining the purpose and function of the broad, international field of health education, and (b) developing curriculum standards within professional preparation programs for the purpose of strengthening the training of health education students seeking various distinct health-related occupations. Future e-Delphi work to establish this consensus would ideally capture and share such input from distinguished health educators not just in the United States, but around the world. The e-Delphi study, in and of itself, could uniquely and effectively connect the opinions of expert health education researchers and practitioners from around the world, and provide a research forum to systematically generate an agreed upon direction for the future of the profession. The findings from such a technologically-mediated e-Delphi study might well mark a watershed moment for the field, one which could not have been conceptualized or coordinated without this kind of research technology. Given the potential of the e-Delphi study to address these kinds of research questions, it is worth distinguishing those numerous benefits of the e-Delphi which bear noting.
Convenience is a benefit for both the e-Delphi administrator and the research participants. The administrator can log-in to the 'virtual laboratory' at any time to design, adjust, or monitor any number of simultaneous activities. For example, the administrator can chose a level of anonymity whereby participants may be entirely anonymous or they may be revealed to others through a user profile (containing information on location or expertise). The administrator can also control access to information so that participants may only be able to access their own responses or they may be allowed to see what others have contributed, which is integral to facilitating fidelity to the methodological process. In contrast to the traditional Delphi, e-Delphi research participants can access the virtual laboratory wherever there is Internet access and when it is most convenient for them to do so. The process therefore occurs in 'real-time', allowing the administrator and participants to remain conveniently connected and up to date as the e-Delphi progresses.
Time and cost savings are reported to be the most persuasive advantage of Internet-based research. (49) Day and Bobeva (50) report that the e-Delphi is an increasingly attractive alternative for mitigating the traditionally long waits between Delphi iterations when surface mail is the primary communication channel. The time associated with the conventional Delphi is identified not only as a cause of attrition specifically, but also a deterrent to Delphi research (for both researchers and participants) generally. In this regard, Donohoe and Needham (45) claim that the use of the Internet and the time savings it permits enhances the appeal of using Delphi methodology and for participating in Delphi research. De Villiers et al. (51) report that the major advantage of the e-Delphi is the cutting of costs, time and effort as it enables a group of experts to be canvassed rapidly and inexpensively (eliminating printing and postage costs) without geographical limitations. They conclude that the technique's suitability for electronic administration makes it cost-effective and efficient to use for health education research generally and for international health education research explicitly. This advantageous trait is corroborated by Atkinson and Gold (52) who note that the e-Delphi facilitates a cost-efficient and practical way to use consensus-building strategies without the distance constraints imposed by geography.
Group research usually requires significant data management investments. Keeping track of personal information and managing data collection and analysis are common challenges. (49) In a survey of qualitative researchers, Miles and Huberman (53) reported that 75% are using electronic data management software, while Mann and Stewart (49) reported an increased use because new software offers several advantages. It facilitates data management, reduces risk of error and makes the analytic process more transparent. In the burgeoning era of cloud computing, researchers in health education should strongly consider the important security and access benefits that research endeavours such as the e-Delphi enable. Such benefits to data management in research are sure to proliferate with time; therefore, health educators should considering adopting these Web-based software and storage technologies when designing and conducting e-Delphi studies.
In light of these evident advantages, a literature search was conducted to determine the extent of static inquiry done using the Internet to conduct Delphi research. A Web of Science[R] bibliographic database search using the keywords 'Delphi' and 'Internet' produced 79 peer reviewed articles published between 1988 and 2010. Of these, 11 were unrelated because they focused on a computer software program of the same name--Delphi. Of the remaining 68 articles, 20 applied a traditional Delphi to the study of the Internet (54,55,56) and 46 reported the results of an e-Delphi study (all published after 1998). In the 46 articles where the Internet was used to support, facilitate, or enable Delphi inquiry, the primary Internet function was as an email portal through which surveys were delivered and responses were collected. (57,58,59) Two articles report using web pages for posting and sharing Delphi reports and resources with panellists, three report using online conference technology for panel communication, and twenty use electronic survey software (e-survey) for collecting, managing and analysing data (47,60-64)
Figure 1 depicts the linear progression of e-Delphi publications in peer-reviewed scientific journals over the last 10+ years. It is important to remember that while the total number of e-Delphi's is small compared to the number of traditional Delphi's published in the scientific literature during the same time period, the number of researchers using the Internet for Delphi administration has been steadily increasing since 1997. The growing interest in reporting e-Delphi (versus traditional Delphi) results is evidence of its benefits. Importantly, however, only five studies have contributed any methodological critique (65,66) of the e-Delphi. This illustrates that manuscripts on e-Delphi design and administration processes are consistently missing from the literature.
[FIGURE 1 OMITTED]
e-Delphi Limitations
Donohoe and Needham (45) recognize that using the Internet for Delphi research presents a new and exciting research frontier, but they caution that deficiencies in methodological discourse is putting at risk the design, implementation, success and evolution of e-Delphi research. Clearly, this is an important and evolving contribution area for Delphi research specifically, and Internet-based research generally. Despite the reported advantages of the e-Delphi, the review of the literature described above suggests that methodological limitations may be manifesting and precluding widespread implementation of the research technique. Internet accessibility challenges, technological difficulties and the inconvenience of entering data into computer-based data screens over the convenience of hard copy 'page-flipping' (note: this is also reported as an advantage) are presenting new and unforeseen challenges. While many of the issues can be considered characteristic of computers and the Internet, problems associated with traditional methods can be intensified by the conditions of the virtual landscape. These include perceived anonymity, respondent identity (real or perceived) and data accuracy (response selection control and transmission errors). (67,68,69) Because of the problems associated with e-research, select issues--access and control--are brought forward for discussion specifically as they pertain to e-Delphi research in health education.
Access
The literature establishes that the Internet is a tool for extending the potential research population or sample but it is also assumes that people have access to the Internet. (49) The United Nations International Telecommunications Union (ITU) reported that the number of Internet users surpassed 2 billion in 2010 and projected that half of the world's population will have broadband access by 2015. (70) While the United States has historically maintained the most users on a per country basis, growth is not limited to developed nations. In fact, the spatial reach or 'penetration rate' of the Internet continues to grow within developing countries as well as increasingly within remote corners of the world. China, for example, surpassed the United States in 2006 with the most Internet users--160 million, as a result of their rapidly improving infrastructure and growing economy. (70) Economic and infrastructure growth, however, has failed to thwart slow Internet penetration rates and access constraints in remote, rural and developing areas. Bernhardt and Hubley. (17) caution that despite the promise and potential, many significant obstacles remain to be overcome in health education research including, but not limited to, the vast divide that exists between those with access to Internet technologies and those who are unlikely to ever have such access.
Beyond infrastructure, unreliable Internet access can also present a significant challenge for e-Delphi administrators and participants. Where 'dial-up' access is the norm or where access is interrupted by weather, controlled by political interventions, or terminated by acts of war or terrorism, the e-Delphi can be significantly affected. The challenge of sustaining electronic communications with participants is magnified by these constraints--particularly when conducting health research in rural, Indigenous, or impoverished communities. Access to mobile networks is becoming more readily available, however, with coverage in over 90% of the world's population and 143 countries offering 3G services. (70) Unfortunately, Internet access remains expensive, especially in low-income developing countries such as several Latin America nations and Africa. (70) Nevertheless, studies are now being done to develop and refine open source, standards-based tools for mobile data collection, aggregation, analysis and reporting in these types of low income regions of the world. (73)
To maximize communications and time-savings as well as to mitigate these potential limitations, it is recommended that the researcher offer a pen-and-paper survey alternative, invest in developing clear directions for survey completion, and pilot test Internet-based surveys and communications (to avoid simple technological errors). It is also highly recommended that the e-Delphi administrator make themselves or an assistant available for technical support throughout the duration of a Delphi exercise (many e-survey providers offer phone and Internet support). To enable research networks using the Internet, Bernhardt and Hubley (17) and Donohoe (18) constructed websites for sharing research information (a central site from which all information, surveys and resources could be accessed), used a browser-friendly survey delivery system, pilot-tested the surveys and delivery mechanisms, and provided technological support (for participants) to mitigate access issues. They report that these research design and administration decisions facilitated a generally seamless e-Delphi process.
Research Control
It has been suggested that control of the 'virtual laboratory' is more difficult than the traditional research laboratory. Concerns related to false representation, lack of verbal interaction, and participant distractions have been reported in the Internet inquiry literature. (49,74) In the case of the e-Delphi, the research environment is 'virtual' regardless of the medium of communication, with verbal and physical interaction not acting as a distinguishing feature of the method. However, control issues can be magnified when using the e-Delphi over the traditional Delphi method (where the Delphi survey is delivered by post to the expert respondent).
The Delphi method, by its very nature, is highly selective about the type of respondents that are invited to participate, and this is also true in health education research. (51,52) The Delphi relies on opinion and this is a distinguishing feature and methodological absolute. While the literature details a number of ways in which expertise can be assessed and/or established (e.g. self-identification, screening survey, third party verification (45)), the anonymity of the Internet presents concerns related to representation. In effect, when using the Internet to conduct research, 'there is a high degree of uncertainty ... in terms of 'knowing' the identity of the other' and this may raise concerns when seeking Institutional Review Board (IRB) approval for e-Delphi research. (75(p.5)) To mitigate potential control limitations and to address potential IRB concerns, it is recommended that a secure hyperlink be provided to expert participants (unique and accessed only by each individual participant--so as not to be forwarded or opened by others) along with passwords (in a separate email preferably) to access the e-surveys (see Table 1). Bernhardt and Hubley (17) report that this practice provides reassurance that the survey respondent was indeed the invited expert participant. However, this method is not foolproof and control remains an issue yet unresolved in Internet-based inquiry. Further research must be conducted to better understand how to control the virtual laboratory and avoid representation concerns. By extension, researchers are encouraged to share their e-Delphi methodological experiences with the health education research community so that best-practices can evolve.
On the basis of the advantages and limitations discussed here, Table 1 introduces a set of mitigation measures to assist e-Delphi architects and administrators in overcoming Internet access and experimental control impediments.
Maximizing the Potential of the e-Delphi Technique
Table 2 presents a set of recommendations to address the most pressing limitations of the traditional Delphi by capitalizing on the potential advantages of Internet-based tools and techniques.
When using the Delphi technique in health education, it is recommended that the researcher critically assess the advantages and limitations of Internet-based tools before making experimental design decisions. Should an e-Delphi be selected as the methodological best-fit for the study, an Internet-based research portal should be established to legitimize the study as well as to serve as the primary communication medium between the researcher(s) and research participants. Information about the study (purpose, objectives, timelines, ethics, etc.), contact information, password protected e-survey links and Delphi round reports can be posted to ensure transparency throughout the research process. Before selecting an e-survey software or service provider (e.g., Qualtrics, SurveyMonkey), several operational factors should be considered, such as: design features and interface (i.e., ease of use, accessibility, etc.), level of respondent access and technological support, and options for data management and analysis. Many services now offer an upgrade option that provides access to additional data storage as well as advanced data analysis tools (e.g. response tracking, attrition monitoring, and statistical reporting). Throughout the e-Delphi process, it is also highly recommended that the researchers maintain up-to-date communications with the research participants. This can be accomplished through email and through regular updates to the Internet-based research portal--so as to mitigate attrition and to keep on track with established timelines.
Additionally, the use of mobile and smart phone technology to collect data during e-Delphi studies is an important area to investigate in the literature. Mobile Internet usage is growing rapidly worldwide, making real-time information tools more readily available to clinicians, practitioners and the public. (76) Early efforts made to tap the power of mobile software tools to engage research participants in the data collection process will likely represent an important step in improving health education research. With greater adoption of mobile and smartphone technology for data collection, there will likely be a set of new methodological e-Delphi issues to consider, particularly related to the previously cited limitations of access to the Internet, control over participant access to survey instruments, and other related consumer health informatics issues impacting mobile device usability. Future research should investigate how access to the Internet on mobile devices impacts the e-Delphi with regards to access and control over the research process.
CONCLUSION
Internet-based technologies are without a doubt, transforming research. The traditional grounds and audiences for research are shifting; (67) innovative data capture and knowledge sharing methods are developing; (46) space-time relations and geographical boundaries are changing; (77) and the kinds of research issues that are being explored are evolving. (68) Whether the Internet is being used as a portal for health education or as a channel for conducting health education research, it is clear that the Internet can have a significant impact on the ways in which health education research and practice are conducted in the future. (17)
In particular, the e-Delphi offers unparalleled convenience, time and cost savings, and data management options. Although the e-Delphi technique is a creative and efficient method for facilitating health education research, there are limitations associated with the Delphi study in this relatively new 'virtual landscape.' While many of the issues can be considered to be a reflection of the nature of technology and the Internet, some limitations associated with the traditional Delphi can be intensified and new issues can present when the Internet is used to enable the technique. It must also be acknowledged that the e-Delphi and other Internet-based approaches are simply tools to be used and applied, when appropriate, within the context of health education theories, values, principles and models. At this time, additional research is needed to evaluate the effects of Internet-based research tools such as the e-Delphi for health education and to explore its effects relative to traditional educational approaches. Nonetheless, by adhering to the suggestions for overcoming stated methodological obstacles, the health education researcher can make the most of this most promising e-research technique. In doing so, researchers in health education can make strong contributions to the evolution of methodological best-practices for this most relevant, consensus-building technique.
Event Code: 1-2012
For Continuing Education Contact Hours Article
Advantages and Limitations of the e-Delphi Technique: Implications for Health Education Researchers Congruent to Responsibility IV: Conduct Evaluation and Research Related to Health Education
1. Over--indexed webpages can be identified using common search engines such as Yahoo!, Google, Bing and Ask.
a. 9 billion
b. 10 billion
c. 11 billion
d. 13 billion
2. --percent of wireless Internet users have looked online for health information compared to 70% of Internet users who do not use a mobile connection.
a. 78
b. 81
c. 83
d. 91
3. In a study by Benfield and Szlemko it was found that not until the last decade that the use of Internet for primary data collection began to increase.
a. true
b. false
4. --are used by 74.6 million people in the U.S. and the penetration rate is even higher in Asia and the Pacific regions of the world.
a. Desktop computers
b. Notebook computers
c. Smartphones
d. Modems
5. There is a paucity of-- which exist to direct Internet-based research activities in the health professions.
a. practice based guidelines
b. websites
c. urls
d. servers
6. The Delphi Technique was originally developed by the
a. Army.
b. RAND Corporation.
c. CIA.
d. automobile industry.
7. The Delphi is a method for organizing conflicting values and experiences and facilitates the incorporation of multiple opinions into
a. consensus.
b. nodes.
c. channels.
d. themes
8. Donohoe and Needham claim that Delphi's popularity may be attributed to
a. Flexibility
b. Reflexivity
c. Proximity
d. all of the above
9. The conduct of Delphi studies is amenable to the Internet platform where --collection of data can be made more efficient.
a. common
b. infrequent
c. iterative
d. daily
10. In a survey of qualitative researchers, Miles and Huberman reported that 75% are using
a. pencil and paper data collection.
b. video recorders.
c. audio recorders.
d. electronic data management software.
Submitted May 1, 2011. Accepted July 14, 2011.
REFERENCES
(1.) de Kunder M. The size of the world wide web. 2011. Available at: http://www.worldwidewebsize.com. Accessed January 31,2011.
(2.) Gaudin S. Worldwide poll: 4 of 5 call Internet access a basic human right. Available at: http://www.techworld.com.au/article/ 3 39011 / worldwide_ poll_4_5_call_internet_access_basic_human_right/. Accessed June 15, 2011.
(3.) Fox S. Health topics: 80% of internet users look for health information online. Washington, DC: Pew Internet & American Life Project; 2011.
(4.) Jackson EL. Leisure and the Internet. JOPERD--The Journal of Physical Education, Recreation & Dance. 1999;70:18.
(5.) Leung L, Lee R Multiple determinants of life quality: the roles of Internet activities, use of new media, social support, and leisure activities. Telematics and Informatics. 2004;22:161-180.
(6.) Bargh J, McKenna K. The Internet and social life. Annu Rev Psychol. 2004;55:573-590.
(7.) Engelberg E, Sj6berg L. Internet use, social skills, and adjustment. CyberPsychol Behav. 2004; 7: 41-47.
(8.) Fogel J, Albert S, Schnabel F, et al. Internet use and social support in women with breast cancer. Health Psychol. 2002; 21:398-404.
(9.) Kraut R, Kiesler S, Boneva B, et al. Internet paradox revisited. J Soc Issues. 2002; 58:49-74.
(10.) Quan-Haase A, Wellman B. How Does the Internet affect social capital? In Huysman M, Wulf V, eds. Social Capital and Information Technology. Massachusetts: Massachusetts Institute of Technology, 2004:113-131.
(11.) Ellison NB, Steinfield C, Lampe C. The benefits of facebook 'friends': Social capital and college students use of online social network sites. Journal of Mediated Communication. 2007;12:1143-1168.
(12.) Cassell M, Jackson C, Cheuvront B. Health communication on the Internet: An effective channel for health behaviour change? J Health Commun. 1998;3:71-79.
(13.) Eysenbach G. What is e-health? J Med Internet Res. 2001; 3(2): E20.
(14.) Korp P. Health on the Internet: implications for health promotion. Health Educ Res. 2005;21(1):78-86.
(15.) Escoffery C, Miner KR, Adame D, et al. Internet use for health information among college students. J Am Coll Health. 2005;53:183-188.
(16.) Stellefson M, Chaney B, Chaney D. The digital divide in health education: Myth or reality? American Journal of Health Education. 2008;39:106-112.
(17.) Bernhardt J, Hubley J. Health education and the Internet: the beginning of a revolution. Health Educ Res. 2001;16:643-645.
(18.) Donohoe HM. Internet-based leisure research: Opportunities and constraints--Pesquisa no lazer associados a Internet: Oportunidades e entraves. Motriz. Revista de Educaao Fisica, UNESP. 2008; 14:1-8.
(19.) Montelpare WJ, McPherson MN. Data processing across the internet: A model for design. The International Electronic Journal of Health Education. 1999;2:127-137.
(20.) Suarez-Balcazar Y, Balcazar FE, Taylor-Ritzler. Using the Internet to conduct research with culturally diverse populations: Challenges and opportunities. Cultur Divers Ethnic Minor Psychol. 2009;15:96-104.
(21.) Benfield JA, Szlemko WJ. Internet-based data collection: Promises and realities. Journal of Research Practice. 2006;2:D1.
(22.) Birnbaum MH. Psychological Experiments on the Internet. New York: Academic Press; 2000.
(23.) Duffy ME. Methodological issues in web-based research. J Nurs Scholarsh. 2002; First Quarter: 83-88.
(24.) Ginossar T, Nelson S. Reducing the health and digital divides: A model for using community-based participatory research approach to e-health interventions in low-income Hispanic communities. Journal of Computer-Mediated Communication. 2010;15:530-551.
(25.) Schmidt WC. World-wide web survey research: benefits, potential problems and solutions. Behav Res Methods. 1997;29:274-279.
(26.) Waltz CF, Strickland O, Lenz ER. Measurement in Nursing and Health Research. New York, NY: Springer Publishing Company; 2010.
(27.) Witmer D, Colman R, Katzman S. From paper-and-pencil to screen-and-keyboard: Toward a methodology for survey research on the Internet. In Jones S, ed. Doing Internet Research: Critical Issues and Methods for Examining the Net. Thousand Oaks, California; Sage Publications; 1999:145-162.
(28.) Wright KB. Researching Internet-based populations: advantages and disadvantages of online survey research, online questionnaire authoring software packages, and web survey services. Journal of Computer-Mediated Communication. 2005;10: article 11.
(29.) O'Connor H, Madge C. 'Focus groups in cyberspace': Using the Internet for qualitative research. Qualitative Market Research: An International Journal. 2003; 6:133-143.
(30.) Moloney MR Dietrich AS, Strickland O, Myerburg SJD. Using Internet discussion boards as virtual focus groups. ANS Adv Nurs Sci. 2003;26: 274-286.
(31.) Underhill C, Olmsted MG. An experimental comparison of computer-mediated and face-to-face focus groups. Social Science Computer Review. 2003;21:506-512.
(32.) comScore, comScore Reports December 2010 U.S. Mobile Subscriber Market Share, 2011. Available at: http://www.comscore.com/ Press_Events/Press_Releases/2011/2/comScore_ Reports_December_2010_U.S._Mobile_Subscriber Market_Share.
(33.) International Telecommunication Union. Key Global Telecom Indicators for the World Telecommunication Service Sector. Available at: http://www.itu.int/ITU-D/ict/statistics/ at_glance/KeyTelecom.html.
(34.) Liamputtong E Qualitative cyber research: An introduction. In, Liamputtong Reds. Health Research in Cyberspace. New York: Nova Science Publishers;2006.
(35.) de Meyrick J. The Delphi method and health research. Health Education. 2003; 103:7-16.
(36.) Linstone HA, Turoff M. The Delphi Method: Techniques and Applications. London: Addison-Wesley Publishing Company, Inc.: 1975.
(37.) Landeta J. Current validity of the Delphi method in social sciences. Technological Forecasting and Social Change. 2006;73:467-482.
(38.) Laflin MT, Horowitz SM, Nims JK. A researcher's guide to health education journals. Am J Health Behav. 1999;23:210-232.
(39.) Frazer GH, Kush R, Richardson CE. Research questions in health education: a professional evaluation. J Sch Health. 1984;54: 188-192.
(40.) Ratnapradipa D, Brown S, Wodika A. Examining the breadth and depth of environmental health through a modifed Delphi technique. Am J Health Educ. 2011;42:50-57.
(41.) Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311:376-380.
(42.) Powell C. The Delphi technique: Myths and realities. J Adv Nurs. 2003;41:376-382.
(43.) Neutens JJ, Rubinson L. Research Techniques for the Health Sciences. Benjamin Cummins: San Francisco, CA; 2009.
(44.) Erfmeyer RC, Erffmeyer ES, Lane IM. The Delphi technique: An empirical evaluation of the optimal number of rounds. Group and Organization Studies. 1986; 11:120-128.
(45.) Donohoe HM, Needham RD. Moving best practice forward: Delphi characteristics, advantages, potential problems, and solutions. International Journal of Tourism Research. 2009;11:415-437.
(46.) MacEachren AM, Pike W, Yu C, et al. Building a geocollaboratory: supporting human-environment regional observatory (HERO) collaborative science activities. Computers, Environment and Urban Systems. 2006;30:201-225.
(47.) Deshpande AM, Shiffman RN, Nadkarni PM. Meta-driven Delphi rating the Internet. Comput Methods Programs Biomed. 2005;77:49-56.
(48.) Stellefson M L, Barry A, Chaney BH, et al. Establishing cause for developing and implementing a broad-based marketing effort for the health education profession. Health Promot Pract. 2011;12:379-387.
(49.) Mann C, Stewart F. Internet communication and qualitative research: a handbook for researching online. London: Sage Publications; 2000.
(50.) Day J, Bobeva M. A generic toolkit for the successful management of Delphi studies. The Electronic Journal of Business Research Methodology. 2005;3:103-116.
(51.) de Villiers M, de Villiers R Kent A. The Delphi technique in health sciences education research. Med Teach. 2005;27:639-643.
(52.) Atkinson N, Gold R. Online research to guide knowledge management planning. Health Educ Res. 2001; 16:747-763.
(53.) Miles M, Huberman A. Qualitative Data Analysis: A Sourcebook of New Methods. Thousand Oaks, CA: Sage; 1994.
(54.) Crutzen R, de Nooijer J, Brouwer W, et al. Internet-delivered interventions aimed at adolescents: a Delphi study on dissemination and exposure. Health Educ Res. 2008; 23:427-439.
(55.) Du YE Librarians' responses to 'reading at risk': A delphi study. Library and Information Science Research. 2009; 31:46-53.
(56.) McBride A, Pates R, Ramadan R, McGowan C. Delphi survey of experts' opinions on strategies used by community pharmacists to reduce over-the-counter drug misuse. Addiction. 2003; 98:487-497.
(57.) Brouwer W, Oenema A, Crutzen R, et al. An exploration of factors related to dissemination of and exposure to internet-delivered behaviour change interventions aimed at adults: A Delphi study approach. J Med Internet Res. 2008; 10:El0.
(58.) Marsden J, Dolan B, Holt L. Nurse practitioner practice and deployment: electronic mail delphi study. J Adv Nuts. 2003; 43:595-605.
(59.) Oestergaard S. Anticipated outcomes from introduction of 5-HTTLPR genotyping for depressed patients: An expert Delphi analysis. Public Health Genomics. 2010; 13:406-414.
(60.) Banks D, Shi R, McLarty J, et al. American college of chest physicians consensus statement on the respiratory health effects of asbestos results of a Delphi study. CHEST. 2009; 135:1619-1627.
(61.) Elwyn G, O'Connor A, Stacey D, et al. Developing a quality criteria framework for patient decision aids: Online international Delphi consensus process. BMI. 2006;333:417-419.
(62.) Li Y. A web-enabled hybrid approach to strategic marketing planning: Group Delphi plus a web-based expert system. Expert Systems with Applications. 2005;29:393-400.
(63.) Schieman C, Kelly E, Gelfand G, et al. Thoracic surgery training in Canada according to the residents: The thoracic surgery resident survey, of the Canadian thoracic manpower and education study (T-Med). Journal of Surgical Education. 2010;67:325-334.
(64.) Tjia J, Field TS, Garber LD, et al. Development and pilot testing of guidelines to monitor high-risk medications in the ambulatory setting. Am J Manag Care. 2010;16:489-496.
(65.) Brown S, Greengrass M. Research portals in the arts and humanities. Literary and Linguistic Computing. 2010;25:1-21.
(66.) Snyder-Halpern R, Thompson CB, Schaffer J. Comparison of mailed vs. internet applications of the Delphi technique in clinical informatics research, l Am Med Inform Assoc. 2000;809-813.
(67.) Baym NK, Markham AN. Making smart choices on shifting ground. In, Markham A, Baym N, eds. Internet Inquiry: Conversations about Method: vii-xix. Thousand Oaks, CA: Sage; 2009.
(68.) Jones S. Studying the net. In Jones S, eds. Doing Internet Research: Critical Issues and Methods for Examining the Net. Thousand Oaks, California; Sage Publications; 1999: 1-28.
(69.) Roztocki N. Using Internet-based surveys for academic research: Opportunities and problems. American Society of Engineering Management (ASEM) National Conference Proceedings. Huntsville, AL, USA, 2001. Available at: http:// www2.newpaltz.edu/~roztockn/alabam01.pdf. Accessed February 4, 2011.
(70.) International Telecommunication Union. ITU Estimates Two Billion People Online by end 2010. Available at: http://www.itu.int/net/ pressoffice/press_releases/2010/39.aspx. Accessed June 15,2011.
(73.) Anokwa Y, Hartung C, Lerer A, et al. A New Generation of Open Source Data Collection Tools. Information and Communication Technologies and Development (ICTD), 2009 International Conference on Date: 17-19 April 2009.
(74.) Orgard S, Bakardieva M, Gajjala R. How can researchers make sense of the issues involved in collecting and interpreting online and offline data? In Markham A, Baym N, eds. Internet Inquiry: Conversations about Method. Thousand Oaks: Sage Publications Inc.; 2009: 33-68.
(75.) Ward K.I. The cyber-ethnographic (re) construction of two feminist online communities. Sociological Research Online. 1999;4. Available at: http://www.socresonline.org.uk/4/1/ ward.html. Accessed June 22, 2011.
(76.) Freifield C, Chunara R, Mekaru SR, et al. Participatory epidemiology: use of mobile phones for community-based health reporting. PLoS Medicine. 2010;7: e1000376.
(77.) Kitchin RM. Towards geographies of cyberspace. Progress in Human Geography. 1998;22:385-406.
Holly Donohoe is an assistant professor in the Eric Friedheim Tourism Institute, University of Florida, FLG 306, Gainesville, FL 32611; E-mail: hdonohoe@hhp.ufl.edu. Michael Stellefson is an assistant Professor in the Department of Health Education and Behavior, University of Florida, FLG 8, Gainesville, FL 32611. Bethany Tennant is a doctoral student in the Department of Health Education and Behavior, University of Florida, Gainesville, FL 32611.
Table 1. e-Delphi Limitations and Mitigation MeasuresLimitations Mitigation MeasuresInternet Access and * Make traditional pen-and-paperTechnology survey available to those without access or to those who experience difficulties * Maintain copies of all documents, data, and reports in case of hardware failure * Offer financial assistance for those without access or without reliable access (cost to use Inter net service at library, cafe or other) * Select an established Internet-based survey provider (consult the literature for guidance) with help features for survey designers and respondents * Downloadable guides and' real-time chat' are particularly valuable * Provide clear instructions to participants and consider developing a 'frequently asked questions' or 'how-to guide' * Provide technical support through a variety of mediums (e.g., email, phone, Skype) for the duration of the exercise * Pilot test all communications and surveys to avoid interpretation and technological difficulties, then test againExperimental Control * Identify known or possible distractions and time administration threats accordingly. E.g. avoid vacation periods, major conferences, or 'high-season' times * Invest in strategies to avoid misrepresentation * Create a screening tool to capture and winnow out the most appropriate experts for the study * Corroborate self-identified experts through membership organizations, academic or professional institutions, publications, etc. * Ensure secure survey access (unique passwords and hyperlinks)Table 2. Traditional Delphi Limitations and e-Delphi RecommendationsLimitations RecommendationsDesign Sensitivity * Critically assess the advantages and limitations of Internet-based tools before making design decisions * Consult Delphi and Internet-based research literature for guidanceRecruitment * Make use of Internet-based databases and communications to identify and recruit potential participants * Establish firm timelines and make them clear to potential participants * Establish a research portal to share information to legitimize the Delphi studyTime Commitment * Use an Internet-based survey platform to reduce time required for postal surveys and communications * Set firm deadlines, provide deadline reminders and adhere to them * Consider investing in Internet-based survey platform upgrades to enable advanced information sharing * Make use of Internet-based survey platform data processing tools such as response tracking and statistical reportingAttrition * Select participants with high interest in the research problem and/or results * Inform participants of the process and goals at the outset * Invest in transparency--share information and results on research web portal * Maintain communications with participants throughout exercise and provide regular timeline reminders * Monitor attrition through Internet-based survey platformConsensus * Different rules of thumb are considered equally valid--consult the literature for guidance standard deviation is commonly used to access consensus * Construct Internet-based survey tool to include a consensus monitoring function * Lack of consensus must be considered a valuable finding
