Background: Across the board, stakeholders in the pharmaceutical product development and approval process have recognized the importance of recruiting diverse patient populations in clinical trials. For example, over the past few decades, Food and Drug Administration (FDA) policies and guidance have aimed to promote practices that lead clinical trials to be more representative of the population(s) most likely to use the product once it is approved1-5. More recently, the FDA published draft guidelines in June 2019 that provide insights about enhancing the diversity of clinical trial populations in terms of eligibility criteria, enrollment practices and trial design6. Researchers, consumer advocacy groups and advocates from the industry have also been actively working toward making clinical trial populations more diverse. To help make recruitment of diverse patient populations for clinical trials a reality, the Society for Clinical Research Sites (SCRS) created the Diversity Awareness Program7, whose aim is to help promote awareness of diversity-related recruitment opportunities via various mechanisms including white papers8. Through the program, SCRS has not only helped sites explore and understand the factors that drive successful recruitment of diverse patient populations but also created an assessment tool to guide sites in improving their ability to recruit diverse patient populations. Between 2018 and 2019, members of a diversity working group led by SCRS developed a 27-item checklist that consisted of statements representing best practices in recruitment of diverse patient populations for clinical trials. This checklist was meant to be used by sites to self-assess their own practices and develop an action plan for improvement. It was also important to examine and evaluate the psychometric properties of the checklist before sites used it for self-assessment.

This paper describes the efforts made toward the development and psychometric testing of the diversity assessment tool.

Objective: This pilot study aimed to examine the psychometric properties of a checklist developed to help clinical trial sites self-assess the extent to which they use best practices for recruiting diverse patient populations for clinical trials.

Methods: In Fall 2019, a mixed-methods approach was used to examine the interpretability and content validity of the checklist. The checklist (Appendix A) was used to develop a survey, and a purposive cohort of 10 clinical site members were invited to complete it. The survey asked respondents to read each statement on the checklist and evaluate whether the statement belonged on the checklist and whether it was appropriately worded. After each respondent completed the survey, they participated in a debrief interview that used the think aloud strategy9 to explore the cognitive processes they used with regard to interpretability and reliability of the statements. Using the survey data, descriptive statistics were used to calculate the agreement that the statement was an element of the best practices for diverse patient recruitment in a clinical trial. Data from the debrief interviews using the think aloud strategy were used to make edits to the statements for which agreement was poor and where there was a concern that the interpretability of the statement would affect its reliability.

Results:
Part I: Survey
Number of Participants Surveyed: 10

As the table below indicates, five out of the 27 items (90% agreement; those with no shading) had near unanimous agreement that they belong to the best practices assessment checklist as-is. Another 16 (above 66% agreement; those in yellow shading) had a majority agreement that they should be included. A total of six items (below 66% agreement; those in red shading) did not have optimal agreement about inclusion in the best practices assessment checklist as they were originally written.

Checklist Statement Agreement that the item is an element of the best practices 
1. Our site knows the demographic make-up of our community. 90% (9 out of 10)
2. Our site knows the demographics of the population we serve. 80% (8 out of 10)
3. Site tracks the actual patient demographics of the clinical trial participants. 80% (8 out of 10)
4. Investigator, Study Coordinator, and/or support staff represent the
patient population.
60% (6 out of 10)
5. Site has bilingual staff that understands the community colloquialisms and applies them in medical discussions. 70% (7 out of 10)
6. Site has established and communicated diversity goals to staff. 80% (8 out of 10)
7. Site tracks progress toward established diversity goals and has awareness of what drives success. 60% (6 out of 10)
8. Site can provide diversity metrics upon request to sponsor at site evaluation. 90% (9 out of 10)
9. Site has cultural competency and awareness training with staff. 50% (5 out of 10)
10. Site leadership supports the recruitment of diverse patients. 80% (8 out of 10)
11. Site has a process to identify and address barriers. 90% (9 out of 10)
12. Site has an established strategy to approach populations for clinical trials tailored to their target population. 90% (9 out of 10)
13. Site partners with sponsor to create study-specific recruitment plans focused on target patient populations. 70% (7 out of 10)
14. Site creates retention plans focused on target patient populations. 70% (7 out of 10)
15. Site requests appropriate patient materials from sponsor tailored toward diverse populations. 80% (8 out of 10)
16. Site is aware of the predominant languages in your patient population and routinely requests translations from sponsor of all subject-facing materials (e.g., ICF, patient recruitment materials, patient outcomes assessments, E-diaries). 80% (8 out of 10)
17. Site conducts outreach to minority-based organizations to establish a network of referrals (e.g., churches, community centers, food banks, medical community, patient advocacy and support groups, etc.). 70% (7 out of 10)
18. Site utilizes appropriate media outlets (radio, TV, social media) specific to the targeted population. 80% (8 out of 10)
19. Site leverages past experience with recruitment materials to inform future strategies. 80% (8 out of 10)
20. Site has a mechanism to notify patients for eligibility in clinical trials. 80% (8 out of 10)
21. Informed Consent short form is available, pre translated and pre-IRB approved in relevant languages. 60% (6 out of 10)
22. Site accommodates working patients by offering hours of operation outside of normal business hours (i.e., early arrivals and/or after-hours appointments) and weekend visits. 80% (8 out of 10)
23. Site offers phone prompt for other frequently used languages (e.g., press 1 for Spanish). 50% (5 out of 10)
24. Site provides transportation services or requests reimbursement from sponsor as needed. 90% (9 out of 10)
25. Site has agreement with local hotels for providing accommodations for parents/caregivers. 30% (3 out of 10)
26. Stipends are offered and/or distributed in a method easy for patient use and in a timely manner. 80% (8 out of 10)
27. Site accommodates patients with young children. 70% (7 out of 10)

To identify the thought process behind their agreement with the items, robust debrief interviews were conducted with the participants using the think aloud strategy. Results from these interviews are described below.

Part II: Debrief Interviews Using Think Aloud Strategy
Number of Participants Interviewed: 10

The debrief interviews using the think aloud strategy provided an opportunity to discuss why respondents felt the statements were meaningful and appropriate for the checklist and why they were not. Specific comments made during the debrief session for each of the statements on the diversity checklist are given below:

Checklist StatementComments
1. Our site knows the demographic make-up of our community.One person indicated that they think this item is not as well worded as item two.
2. Our site knows the demographics of the population we serve.All those who opined that this item was not necessary indicated that it seems to duplicate item one above. Those individuals suggested eliminating this item.
3. Site tracks the actual patient demographics of the clinical trial participants.One person had an issue with the word “track” and suggested that it be changed to “know” as the site may not track but know off the top of their head.
4. Investigator, Study Coordinator, and/or support staff represent the patient population.The word “represent” is unclear. May want to rewrite it as “culturally equivalent” or something similar.
5. Site has bilingual staff that understands the community colloquialisms and applies them in medical discussions.The word “colloquialisms” may not be clear to everyone. Also, bilingual is not appropriate. May want to simplify this statement to “Site has staff that can utilize commonly used words in the community for medical discussion” or something similar.
6. Site has established and communicated diversity goals to staff.One person indicated that the item would be better received if it was changed to “Staff knows diversity goals of the site”.
7. Site tracks progress toward established diversity goals and has awareness of what drives success.“Has awareness of what drives success” is unclear. Suggested replacing it with “and knows what marketing or outreach strategies work to make them successful”.
8. Site can provide diversity metrics upon request to sponsor at site evaluation.No comments were given.
9. Site has cultural competency and awareness training with staff.Not all sites have formal training. Instead, some have informal mechanisms. Cultural competency needs to be defined. Suggested replacing it with “Site has mechanisms (formal or informal) to make staff culturally aware and sensitive to the patient population”.
10. Site leadership supports the recruitment of diverse patients.It was suggested to change “leadership” to “management”.
11. Site has a process to identify and address barriers.No comments were given.
12. Site has an established strategy to approach populations for clinical trials tailored to their target population.No comments were given.
13. Site partners with sponsor to create study-specific recruitment plans focused on target patient populations.Some sites create their own.
14. Site creates retention plans focused on target patient populations.Two people indicated they were not sure what else the retention plan would be based on if not on target population.
15. Site requests appropriate patient materials from sponsor tailored toward diverse populations.Doesn’t read appropriately. Suggest changing to “Site requests appropriately tailored patient materials from sponsor”.
16. Site is aware of the predominant languages in your patient population and routinely requests translations from sponsor of all subject facing materials (e.g., ICF, patient recruitment materials, patient outcomes assessments, E-diaries).This is asking for several things. Suggest changing to “Site routinely requests materials in languages predominantly used by members of their patient population”.
17. Site conducts outreach to minority-based organizations to establish a network of referrals (e.g., churches, community centers, food banks, medical community, patient advocacy and support groups, etc.).One person felt this is redundant; another indicated that the community they are in is so small that they do not need referrals.
18. Site utilizes appropriate media outlets (radio, TV, social media) specific to the targeted population.One person felt that this is normal practice not best practice.
19. Site leverages past experience with recruitment materials to inform future strategies.Simplify to “Site uses recruitment materials that have worked in the past to decide upon future strategy for recruiting diverse patients”.
20. Site has a mechanism to notify patients for eligibility in clinical trials.One participant was not sure why this should be part of best practice.
21. Informed Consent short form is available, pre-translated and pre-IRB approved in relevant languages.Most participants did not know what an IC short form is. Recommendation was to add the words “when needed” before the statement.
22. Site accommodates working patients by offering hours of operation outside of normal business hours (i.e., early arrivals and/or after hours appointments) and weekend visits.One person indicated that state regulations may interfere and so this may not be feasible.
23. Site offers phone prompt for other frequently used languages (i.e., press 1 for Spanish).One person did not think that this was necessary; others thought that it is now a normal practice but not a best practice.
24. Site provides transportation services or requests reimbursement from sponsor as needed.No comments were given.
25. Site has agreement with local hotels for providing accommodations for parents/caregivers.Several issues: Is it hotels or could it be other accommodations? Is it only parents/caregivers or could this be family members? Suggested rewording to: “If needed, site has provisions for accommodations for patients and their family members”.
26. Stipends are offered and/or distributed in a method easy for patient use and in a timely manner.One person indicated that the timing and nature of stipends are based on contracts and so suggested rewording this item to: “When available, stipends are offered/distributed in a timely and easy to use method”.
27. Site accommodates patients with young children.One person indicated this can be measured under item 25 if it is modified to patients and their family members.
Overall CommentSeveral people opined that it will be important to include a definition of diversity at the start of the checklist as diversity may be about race/ethnicity in most people’s minds, but this checklist considers diversity in the form of gender, race/ethnicity, age and income. Another major comment was that the scale should be expanded from a Yes/No to a more open scale such as Always, Sometimes and Never with some clarity of numbers associated with it.

Based on content expert advisement, the checklist was revised to include the suggested edits and the scale was also revised. The revised version of the checklist is provided in Appendix B.

Conclusions: This pilot study helped identify and resolve issues affecting the interpretability and reliability of responses to statements belonging to the checklist for evaluation of best practices in recruiting diverse patient populations for clinical trials. The validation process helped develop a checklist that is ready for large-scale testing and use by sites for self-assessment of the extent to which they use best practices to recruit diverse patient populations for clinical trials.

Next Steps: In the current study, the Site Assessment Checklist was piloted using qualitative methods and those methods helped establish the face validity and internal consistency of the statements included in the checklist. To further ensure that the developed checklist can be used on a large-scale basis, it needs to be tested among a representative population of site personnel so that quantitative psychometrics (such as a reliability co efficient that establishes the reliability of a scale) and descriptive statistics can further establish its value and utility for identifying best practices for the recruitment of diverse patient populations for clinical trials. This work is in progress with a sample size of 500 subjects.

Sponsorship: The research and development of this scientific research were supported and generously sponsored by Acurian, GSK, Janssen, Lilly, Medidata, Merck, Parexel, PHARMA, Roche, Syneos Health, and VirTrial.

About the Author: Diana Foster, Ph.D., has led the SCRS diversity initiative since its inception in 2016. She had the opportunity to work hand-in-hand with an expert working group representing the pharmaceutical industry, CROs, FDA, 16 | InSite -Summer 2020 clinical research sites, and advocacy and special interest groups. She has held the consulting position of Vice President of Strategy and Development for SCRS for six years and has been integral in developing industry relationships for the organization. This is the third research paper published as a result of the work from this important project.

References:

  1. S.1 — 103rd Congress: National Institutes of Health Revitalization Act of 1993. www.GovTrack.us. 1993. Accessed December 5, 2019, from: https://www.govtrack.us/congress/bills/103/s1
  2. S.830 — 105th Congress: Food and Drug Administration Modernization Act of 1997. www.GovTrack.us. 1997. Accessed December 5, 2019, from: https://www.govtrack.us/congress/bills/105/s830
  3. S.3187 — 112th Congress: Food and Drug Administration Safety and Innovation Act. www.GovTrack.us. 2012. Accessed December 5, 2019, from: https://www.govtrack.us/congress/bills/112/s3187
  4. United States Food and Drug Administration. FDA’s Action Plan to Enhance the Collection and Availability of Demographic Subgroup Data. August 2014. Accessed December 5, 2019, from: https://www.fda.gov/downloads/RegulatoryInformation/Legislation/SignificantAmendmentstotheFDCAct/FDASIA/UCM410474.pdf
  5. United States Food and Drug Administration. 2015-2016 Drug Trials Snapshots Summary Report. 2017. Accessed December 5, 2019, from: https://www.fda.gov/downloads/Drugs/InformationOnDrugs/UCM541327.pdf
  6. United States Food and Drug Administration. Enhancing the Diversity of Clinical Trial Populations—Eligibility Criteria, Enrollment Practices, and Trial Designs [Draft guidance]. Accessed December 20, 2019, from:https://www.fda.gov/media/127712/download
  7. SCRS Launches Diversity Awareness Program. PRNewswire. 2017. Accessed December 20, 2019, from: https://www.prnewswire.com/news-releases/scrs-launches-diversity-awareness-program-300467469.html
  8. Society for Clinical Research Sites. Patient Diversity Awareness: Developing a Better Understanding of the Knowledge, Expertise, and Best Practices at Clinical Research Sites to Meet the Needs of an Increasingly Diverse United States Population [White paper]. Accessed April 4, 2018, from: http://myscrs.org/learningcampus/white-papers/
  9. Aitken LM, Mardegan KJ. “Thinking aloud”: Data collection in the natural setting. Western Journal of Nursing Research 2000; 22:841–853.

Appendix A.

ORIGINAL SITE ASSESSMENT CHECKLIST (Yes/No) FOR EVALUATION OF BEST PRACTICES OF RECRUITMENT OF DIVERSE PATIENT POPULATIONS FOR CLINICAL TRIALS

A. SITE OVERVIEW

  1. Our site knows the demographic make-up of our community.
  2. Our site knows the demographics of the population we serve.
  3. Site tracks the actual patient demographics of the clinical trial participants.
  4. Investigator, Study Coordinator, and/or support staff represent the patient population.
  5. Site has bilingual staff that understands the community colloquialisms and applies them in medical discussions.
  6. Site has established and communicated diversity goals to staff.
  7. Site tracks progress toward established diversity goals and has awareness of what drives success.
  8. Site can provide diversity metrics upon request to sponsor at site evaluation.
  9. Site has cultural competency and awareness training with staff.
  10. Site leadership supports the recruitment of diverse patients.
  11. Site has a process to identify and address barriers.

B. RECRUITMENT AND OUTREACH

  1. Site has an established strategy to approach populations for clinical rials tailored to their target population.
  2. Site partners with sponsor to create study-specific recruitment plans focused on target patient populations.
  3. Site creates retention plans focused on target patient populations.
  4. Site requests appropriate patient materials from sponsor tailored toward diverse populations.
  5. Site is aware of the predominant languages in your patient population and routinely requests translations from sponsor of all subject-facing materials (e.g., ICF, patient recruitment materials, patient outcomes assessments, E-diaries).
  6. Site conducts outreach to minority-based organizations to establish a network of referrals (e.g., churches, community centers, food banks, medical community, patient advocacy and support groups, etc.).
  7. Site utilizes appropriate media outlets (radio, TV, social media) specific to the targeted population.
  8. Site leverages past experience with recruitment materials to inform future strategies.
  9. Site has a mechanism to notify patients for eligibility in clinical trials.

C. PATIENT FOCUSED

  1. Informed Consent short form is available, pre-translated and pre-IRB approved in relevant languages.
  2. Site accommodates working patients by offering hours of operation outside of normal business hours (i.e., early arrivals and/or after-hours appointments) and weekend visits.
  3. Site offers phone prompt for other frequently used languages (e.g., press 1 for Spanish).
  4. Site provides transportation services or requests reimbursement from sponsor as needed.
  5. Site has agreements with local hotels for providing accommodations for parents/caregivers.
  6. Stipends are offered and/or distributed in a method easy for patient use and in a timely manner.
  7. Site accommodates patients with young children.

Appendix B.

REVISED SITE ASSESSMENT CHECKLIST FOR EVALUATION OF BEST PRACTICES OF RECRUITMENT OF DIVERSE PATIENT POPULATIONS FOR CLINICAL TRIALS

Item Always (100% of the time) Some/Most times (50-<100%) Rarely/Never (0-<50%)
A. Site Overview (10 items)
Our site knows the demographic make-up of our community. 5 2.5 0
Our site knows the actual patient demographics of the clinical trial participants. 5 2.5 0
Our site’s Investigators, Study Coordinators, and/or support staff are culturally representative of the patient population. 5 2.5 0
Our site has staff who can use the commonly-used medical terms in our community. 5 2.5 0
Our site staff knows our site’s goals for recruitment of diverse participants. 5 2.5 0
Our site tracks progress toward established goals for recruitment of diverse participants and knows what marketing or outreach strategies work to make them successful. 5 2.5 0
Our site can provide diversity metrics to the sponsor upon request at site evaluation. 5 2.5 0
Our site has mechanisms (formal or informal) to make staff culturally aware and sensitive to the patient population. 5 2.5 0
Our site management team supports the recruitment of diverse patients. 5 2.5 0
Our site has a process to identify and address barriers to recruiting diverse participants. 5 2.5 0
B. Site Recruitment and Outreach (9 items)
Our site has established a tailored strategy to approach targeted populations for clinical trials. 5 2.5 0
When needed, our site partners with the sponsor to create study specific and targeted patient population specific recruitment plans. 5 2.5 0
Our site creates target patient population-specific retention plans. 5 2.5 0
When needed, our site requests appropriately tailored patient materials from the sponsor. 5 2.5 0
Our site routinely requests materials (e.g., ICF, patient recruitment materials, patient outcomes assessments, E-diaries) in languages predominantly used by members of the patient population. 5 2.5 0
When needed, our site conducts outreach to minority based organizations to establish a network of referrals (e.g., churches, community centers, food banks, medical community, patient advocacy and support groups, etc.). 5 2.5 0
When needed, our site utilizes appropriate media outlets (radio, TV, social media) specific to the targeted population. 5 2.5 0
Our site uses recruitment materials that have worked in the past to decide upon future strategy for recruiting diverse patients. 5 2.5 0
Our site has a mechanism to notify patients about their eligibility in clinical trials. 5 2.5 0
C. Patient Focused (7 items)
Our Informed Consent form is available, pre-translated and pre IRB approved in relevant languages. 5 2.5 0
If allowed by regulations, our site accommodates working patients by offering hours of operation outside of normal business hours (i.e., early arrivals and/or after-hours appointments) and weekend visits. 5 2.5 0
Our site offers phone prompts for most frequently used languages (e.g., “Press 1 for Spanish”). 5 2.5 0
Our site provides transportation services or requests reimbursement for travel from sponsor as needed. 5 2.5 0
When needed, our site has provisions for providing accommodations for patients and their family members, including children. 5 2.5 0
Stipends are offered and/or distributed in a timely manner and method that are easy for patients to use. 5 2.5 0

Learn more about the SCRS Diversity Awareness Program visit myscrs.org/learning-campus/diversity-in-clinical-trials/