Chapter 3 Be a QualTRIXter
The focus of this lecture is on the technical and mechanical tools available in Qualtrics (and likely other survey platforms) to increase the effectiveness of your survey.
3.2 Research Vignette
I will demonstrate the qual”TRIX” by using a Qualtrics account hosted at Seattle Pacific University. The only surveys in this account are for the Recentering Psych Stats chapters and lessons. All surveys are designed to not capture personally identifying information and not collecting IP addresses nor longitude/latitude. I use this survey in several lessons in this OER. If you haven’t taken the survey yet, I invite you to do so, now.
As a teaching activity for the ReCentering Psych Stats OER, the topic of the survey was selected to be consistent with the overall theme of OER. Specifically, the purpose of this study is to understand the campus climate for students whose identities make them vulnerable to bias and discrimination. These include students who are Black, non-Black students of color, LGBTQ+ students, international students, and students with disabilities.
After consulting with a diverse group of stakeholders and subject matter experts (and revising the response options numerous times) I have attempted to center anti-Black racism in the U.S. (Mosley et al., 2020, 2021; Singh, 2020). In fact, the display logic does not present the race items when the course is offered outside the U.S. There are only five options for race: biracial/multiracial, Black, non-Black person(s) of color, White, and I did not notice (intended to capture a color-blind response). One unintended negative consequence of this design is that the response options could contribute to colorism (Adames et al., 2021; Capielo Rosario et al., 2019). Another possibility is that the limited options may erase, or make invisible, other identities. At the time that I wrote up the first description of this survey, the murder of six Asian American women in Atlanta had just occurred. The Center for the Study of Hate and Extremism has documented that while overall hate crimes dropped by 7% in 2020, anti-Asian hate crimes reported to the police in America’s largest cities increased by 149% (FACT SHEET, n.d.). These incidents have occurred not only in cities, but in our neighborhoods and on our campuses (P. Kim, 2021; P. Y. Kim, 2021; STOP AAPI HATE, n.d.). While this survey is intended to assess campus climate as a function of race, it unfortunately does not distinguish between many identities that experience marginalization.
Although the dataset should provide the opportunity to test a number of statistical models, one working hypothesis that framed the study is that the there will be a greater sense of belonging and less bias and discrimination when there is similar representation (of identities that are often marginalized) in the instructional faculty and student body. Termed, “structural diversity” (K. R. Lewis & Shah, 2019) this is likely an oversimplification. In fact, an increase in diverse representation without attention to interacting factors can increase hostility on campus (Hurtado, 2007). Thus, the task of rating of a single course relates to the larger campus along the dimensions of belonging and bias/discrimination. For example, if a single class has higher ratings on issues of inclusivity, diversity, and respect, we would expect that sentiment to be echoed in the broader institution.
The survey design has notable limitations You will likely notice that we ask about demographic characteristics of the instructional staff and classmates in the course rated, but we do not ask about the demographic characteristics of the respondent. In making this decision, we likely lose important information. For example, Iacovino and James (2016) have noted that White students perceive campus more favorably than Black student counterparts.
The decision to not collect demographic details about the respondent was about protecting their (your) identity. As you will see, you have the opportunity to download and analyze the data. If a faculty member asked an entire class to take the survey, the date stamp and a handful of demographic identifiers could very likely identify a student. In certain circumstances, this might be risky in that private information (i.e., gender nonconformity, disclosure of a disability) along with course evaluation data and a date stamp could identify the respondent.
Further, the items that ask respondents to guess the identities of the instructional staff and classmates are limited, and contrary to best practices in survey construction that recommend providing the option of a “write-in” a response.
In parallel, the items asking respondents to identity characteristics of the instructional staff along dimensions of gender, international status, and disability are “large buckets” and do not include “write-in” options. Similarly, there was no intent to cause harm by erasing or making invisible individuals whose identities are better defined by different descriptors. Further, no write-in items were allowed. This was also intentional to prevent potential harm caused by people who could leave inappropriate, racist, or otherwise harmful comments.
As I review Qualtrics essentials and trix, I will their use (if used) in the ReCentering Psych Stats survey.
3.3 Qualtrics Essentials
Qualtrics is a powerful program and I find that many of the surveys we distribute don’t capitalize on the features Qualtrics has to offer. Qualtrics has detailed tutorials and instructions that are well worth the investment of a weekend to review them.
In this lecture I will point you to the elements that I think are critical to constructing online surveys. Because Qualtrics tutorials are (a) clear and thorough and (b) frequently updated, I will (a) point you to the tutorials that are available at the time of this lecture prep, (b) tell you why I think they are appropriate, and (c) show you how we have used them in some of our own surveys.
Even if you think you know what you are doing, start here (and then always take the time to “look around” at all the options on each window):
Survey Basic Overview: Qualtrics’ Survey Basic Overview tutorial is a great place to start. From there, you can follow all kinds of leads, looking for things you want to do with your survey – and getting ideas for what will improve it.
Blocks are the basic organizational tool in Qualtrics surveys. Blocks have two purposes: (a) grouping items shown on “one page,” and (b) specifying ordering and/or random selection/presentation in the survey flow.
Question types: Take time to look at all the options. You might be surprised to learn that there is a better choice than you might have imagined.
Let’s take a look at super basic/helpful question types:
- Text/graphic: These are the types you should use for providing information (e.g., informed consent) to the participants or displaying a logo or graphic stimulus.
- Matrix table: The matrix table is a more efficient way to use the Likert-style items (than multiple choice). There is some controversy about whether not to use matrix tables vs. multiple choice dropdowns. As both a survey developer and a respondent, I prefer the matrix table.
- Make sure to select a reasonable amount of header repetitions. This allows the respondent the maximum opportunity to see the column descriptors (and avoid guessing/remembering) while they are responding.
- Slider : The slider is designed for obtaining truly continuous data on a 1 to 100 scale. This range can be adapted to any interval you choose, and you can add anchors to the scale. If the scale you are using is already published, and has not been psychometrically evaluated for slider use, you should probably stick with the format recommended in the publication. But if you are writing test items, consider this option.
- Text Entry Questions: Text boxes have multiple options for answer length.
- Validation: Content validation allows the user to permit certain types of information and specify their formats (e.g., numbers, e-mail addresses, dates). There is art to balancing between being overly restricting and ensuring that the data is entered in the most clear and consistent way possible with honoring the uniqueness of each respondent. Another validation option I frequently use is one that asks individuals if they intended to leave something blank. This is tool that helps prevent missingness without forcing an individual to respond to an item that (a) might not be clear to them, (b) might not be appropriate or them, and/or (c) might result in an answer that is untrue for their unique circumstance.
3.4 Qual-TRIX
Collaborating with other Qualtrics users in your institution is easy! Scroll down to “Collaborating Inside Your Organization” and follow the instructions for adding individuals to your survey (you must “own” the survey; your collaborators will not be able to add others).
The ability to schedule survey distributions is like having your very own assistant! If you have a roster (contact list) you can schedule distributions, reminders, and thank you’s. Qualtrics will keep track of who responds and send reminders to the non-responders. Here are resources for
Personalizing invitations and surveys. Piped text is a way to personalize invitations and/or “carry forward” prior responses into new questions.
Randomization of blocks (or a subset of blocks) can be use for several purposes such as: (a) using random selection to display one or more blocks to respondents – as in a random clinical trial, (b) to randomly display a percentage of blocks or items to shorten the survey in a planned missing design, and (c) randomly display some or all of the blocks of the survey to all respondents so that when respondents experience test fatigue, when they quit responding, “the last items/surveys” aren’t always the same ones. This functions to distribute missingness across surveys.
Randomization of items within a block can be used for similar purposes. You can also use this to display only some of the items (e.g., planned missingness).
File upload from respondents is an additional package that requires the institution to pay a higher fee. If available, this allows respondents to upload some sort of file (photo, PowerPoint, .pdf). We use it for poster contests at professional contests (where students upload their poster for online judging in advance of the conference). A colleague of mine uses this function to collect application elements (i.e., resumes, cover letters, reference letters) to a fellowship program.
- As researchers, we can also upload files (e.g., hardcopy of informed consent, documents to be reviewed) for use by the respondent.
Display, Skip, and/or Branch Logic can be used to help display to respondents only the items that pertain to them. There are multiple approaches to doing this. Using a display logic approach may feel a bit backward where the logic is applied from the landing spot. We did this extensively in as study that involved two language versions and three age options.
Two other approaches for these issues are skip logic and branch logic
3.5 Even moRe, particularly relevant to iRb
We can use Qualtrics tools for purposes beyond collecting and downloading data. These tools are especially useful when I think about IRB applications and ethics related to data collection.
Exporting to Word: Helpful for your IRB application (and perhaps in a cloud so that a team can use track changes to edit), it is super simple to export the survey to Microsoft Word. Additionally, you can specify options for including question numbers, recode values, logic, and so forth. This works well to create a codebook for your research team.
Anonymizing responses: Another step toward an anonymous response is to withhold the IP address, latitude/longitude, and any contact information (e.g., e-mail, name) that you may have uploaded in an e-mail distribution directory. This is accomplished in the Survey Options menu. Do be careful – while anonymizing responses is an ethical, best practice, the deleted information cannot be recovered.
Prevent ballot box stuffing: Want to make sure that respondents only answer once? In the same Survey Options window, you can prevent ballot box stuffing. This is helpful when surveys are distributed with an anonymous link. The tool prevents more than one survey from the same IP address.
Other security options include
- Password protection
- HTTP Referrer verification
Look also at:
- Progress bar to provide participants hope (or despair) for “how much longer.”
- Survey termination to connect custom endings and thank-you notes.
- Partial completion to specify how long the respondent has to complete the survey (after opening it) and whether it is recorded or deleted if it is not completed.
- Related to this, back on the Data & Analysis tab, you can see both the numbers of recorded responses and responses in progress. You also have options to manually determine how you want to include/exclude the responses in progress.
- Failure of the respondent to click the final “–>” submit and progress symbol is often the reason that surveys that are > 90% complete aren’t counted as “complete.” What to do? Options: (a) don’t say “Thanks and goodbye” on a page that has any items, and (b) provide instructions to look for the “–>” symbol to continue.
Finally, PREVIEW PREVIEW PREVIEW! There is no better way check your work than with previews.
3.6 intRavenous Qualtrics
Access credentials for the institutional account, individual user’s account, and survey are essential for getting the survey items and/or results to export into R. The Qualtrics website provides a tutorial for generating an API token.
We need two pieces of information: the root_url and an API token.
- Log into your respective qualtrics.com account.
- Select Account Settings
- Choose “Qualtrics IDs” from the username dropdown
We need the root_url. This is the first part of the web address for the Qualtrics account. For our institution it is: spupsych.az1.qualtrics.com
The API token is in the box labeled, “API.” If it is empty, select, “Generate Token.” If you do not have this option, locate the brand administrator for your Qualtrics account. They will need to set up your account so that you have API privileges.
BE CAREFUL WITH THE API TOKEN This is the key to your Qualtrics accounts. If you leave it in an .rmd file that you forward to someone else, this key and the base URL gives access to every survey in your account. If you share it, you could be releasing survey data to others that would violate confidentiality promises in an IRB application.
If you mistakenly give out your API token, you can generate a new one within your Qualtrics account and re-protect all its contents.
You do need to change the API key/token if you want to download data from a different Qualtrics account. If your list of surveys generates the wrong set of surveys, restart R, make sure you have the correct API token and try again.
# only have to run this ONCE to draw from the same Qualtrics
# account...but will need to get different #token if you are changing
# between accounts.
qualtRics::qualtrics_api_credentials(api_key = "oEwd9qu9xJOf3RoE9iiCZKSs2sfNuSbvy8LnFYxo",
base_url = "spupsych.az1.qualtrics.com", overwrite = TRUE, install = TRUE)
readRenviron("~/.Renviron")all_surveys() generates a dataframe containing information about all the surveys stored on your Qualtrics account.
surveys <- qualtRics::all_surveys()
# View this as an object (found in the right: Environment). Get
# survey id # for the next command If this is showing you the WRONG
# list of surveys, you are pulling from the wrong Qualtrics account
# (i.e., maybe this one instead of your own). Go back and change your
# API token (it saves your old one). Changing the API likely requires
# a restart of R.
surveysTo retrieve the survey, use the fetch_survey() function.
# obtained with the survey ID
#'surveyID' should be the ID from above
#'verbose' prints messages to the R console
#'label', when TRUE, imports data as text responses; if FALSE prints the data as numerical responses
#'convert', when TRUE, attempts to convert certain question types to the 'proper' data type in R; because I don't like guessing, I want to set up my own factors.
#'force_request', when TRUE, always downloads the survey from the API instead of from a temporary directory (i.e., it always goes to the primary source)
# 'import_id', when TRUE includes the unique Qualtrics-assigned ID;
# since I have provided labels, I want false
# Out of the blue, I started getting an error, that R couldn't find
# function 'fetch_survey.' After trying a million things, adding
# qualtRics:: to the front of it solved the problem
QTRX_df <- qualtRics::fetch_survey(surveyID = "SV_b2cClqAlLGQ6nLU", time_zone = NULL,
verbose = FALSE, label = FALSE, convert = FALSE, force_request = TRUE,
import_id = FALSE)
# useLocalTime = TRUE,The optional script below will let you save the simulated data to your computing environment as either a .csv file (think “Excel lite”) or .rds object (preserves any formatting you might do).
# write the simulated data as a .csv write.table(QTRX_df,
# file='QTRX_df.csv', sep=',', col.names=TRUE, row.names=FALSE) bring
# back the simulated dat from a .csv file QTRX_df <- read.csv
# ('QTRX_df.csv', header = TRUE)# to save the df as an .rds (think 'R object') file on your computer;
# it should save in the same file as the .rmd file you are working
# with saveRDS(QTRX_df, 'QTRX_df.rds') bring back the simulated dat
# from an .rds file QTRX_df <- readRDS('QTRX_df.rds')3.6.1 The Codebook
In order to prepare data from a survey, it is critical to know about its content, scoring directions for scales/subscales, and its design. As I demonstrated above, we can export a codebook, that is, a Word (or PDF) version of the survey with all the coding. In Qualtrics the protocol is: Survey/Tools/ImportExport/Export Survey to Word. Then select all the options you want (especially “Show Coded Values”). A tutorial provided by Qualtrics can be found here. This same process can be used to print the PDF example I used above.
I recommend providing custom variable names and recode values directly in Qualtrics before exporting them into R (and before exporting the codebook). A Qualtrics tutorial for this is provided here. In general, consider these qualities when creating variable names:
- Brevity: historically, SPSS variable names could be a maximum of 8 characters.
- Intuitive: although variables can be renamed in R (e.g., for use in charts and tables), it is helpful when the name imported from Qualtrics provides some indication of what the variable is.
- Systematic: start items in a scale with the same stem, followed by the item number – ITEM1, ITEM2, ITEM3.
- Do not include special characters or spaces in variable names; this is problematic for R.
- Do not start variable names with numerals; this is problematic for R.
More complete information about data preparation is covered in chapters in the ReCentering Psych Stats: Multivariate Modeling text.
3.6.2 Using data from an exported Qualtrics .csv file
It is also possible to download the Qualtrics data in a variety of formats (e.g., CSV, Excel, SPSS). Since my R and Qualtrics history began by using files with the CSV extension (think “Excel” lite), that is my preference.
In Qualtrics, these are the steps to download the data: Projects/YOURsurvey/Data & Analysis/Export & Import/Export data/CSV/Use numeric values. In order to import this data into R, it is critical that to save this file in the same folder as the .rmd file that you will use with the data.
R is sensitive to characters used filenames. As downloaded, my Qualtrics .csv file had a long name with spaces and symbols that are not allowed. Therefore, I gave it a simple, sensible, filename, “ReC_Download210319.csv”. An idiosyncrasy of mine is to datestamp filenames. I use two-digit representations of the year, month, and date so that if the letters preceding the date are the same, the files would alphabetize automatically.
# QTRX_csv <- qualtRics::read_survey('ReC_Download210319.csv',
# strip_html = TRUE, import_id = FALSE, time_zone=NULL, legacy =
# FALSE)Although minor tweaking may be required, the same script above should be applicable to this version of the data.
3.6.3 Tweaking Data Format
Two general approaches:
Inside Qualtrics: Use the recode values option (found in the item’s gearbox, to the left of the block) to specify variable names and recode values. These should be preserved on the download.
In the R script: In another lecture I demonstrate how to change the formats of data (character, string), selecting only the variables in which we are interested (e.g., excluding the meta-data), and renaming variables sensibly.
Both work! You can choose your preference. When you are working with a team, map out an explicit process with your collaborators.
3.7 Practice Problems
The suggestion for practice is to develop a questionnaire, format it, pilot it, and download it. Essentially you will be
- Formatting a survey on Qualtrics using all the best practices identified in the lecture. These include:
- Having an introductory statement (to include statement of confidentiality), directions for each sub-survey (if more than one), and a closing statement.
- Selecting the most appropriate question type for the items. For example, matrix instead of multiple choice.
- Within the question type, using the appropriate options for proper formatting (e.g., the anchors in a matrix should be topically consistent and equal interval).
- The survey should include minimum of 3 of the qualTRIXter skills (identified in lecture). Choose from:
- establishing collaboration
- scheduling e-mail distribution and follow-up
- personalizing the survey in some way
- randomization of blocks or items
- integrating display, skip, or branch logic (e.g., having males and females take a different route)
- exporting the survey to Word
- recoding variables in the item controls
- anonymizing the responses
- preventing ballot box stuffing
- including a progress bar
- creating a custom ending, e-mail, or thank-you note
- something else that YOU discovered that isn’t in the lecture
- Piloting it, getting their feedback, and identifying what problems are (and how you might fix them).
- with 3 folks from your research team, cohort, or this class
- with 3 additional folks who aren’t quite as “research savvy”
- collect their feedback (ideally in a text-item directly on the survey itself) and write a brief summary (3 paragraphs max) of their impressions and how you might improve the survey
- Import the Qualtrics data directly R
- preferably, directly from Qualtrics with the API token, base URL, and survey ID
- alternatively (for the same # of points) from the exported CSV file via the qualtRics package (required)
| Assignment Component | Points Possible | Points Earned |
|---|---|---|
| 1. Qualtrics survey best practices | 5 | |
| 2. QualTRIXter skills (at least 3) | 5 | |
| 3. Minimum of 6 pilot respondents | 5 | |
| 4. Summary of pilot feedback | 5 | |
| 5. Import of Qualtrics data into R | 5 | |
| 6. Explanation to grader | 5 | |
| Totals | 20 |