ReCentering Psych Stats: Psychometrics
BOOK COVER
PREFACE
Copyright with Open Access
ACKNOWLEDGEMENTS
1
Introduction
1.1
What to expect in each chapter
1.2
Strategies for Accessing and Using this OER
1.3
If You are New to R
1.3.1
Base R
1.3.2
R Studio
1.3.3
R Hygiene
1.3.4
tRoubleshooting in R maRkdown
1.3.5
stRategies for success
1.3.6
Resources for getting staRted
2
Questionnaire Construction: The Fundamentals
2.1
Navigating this Lesson
2.1.1
Learning Objectives
2.1.2
Planning for Practice
2.1.3
Readings & Resources
2.2
Components of the Questionnaire
2.3
What Improves (or Threatens) Response Rates and Bias?
2.3.1
Should Likert-type scales include a midpoint?
2.3.2
Should
continuous rating scales
be used in surveys?
2.3.3
Should Likert-type response options use an ascending or descending order?
2.3.4
Should surveys include negatively worded items?
2.4
Construct-specific guidance
2.5
Surveying in the Online Environment
2.6
In my Surveys
2.6.1
Demographics and Background Information
2.6.2
Survey Order
2.6.3
Forced Responses
2.7
Practice Problems
3
Be a QualTRIXter
3.1
Navigating this Lesson
3.1.1
Learning Objectives
3.1.2
Planning for Practice
3.1.3
Readings & Resources
3.1.4
Packages
3.2
Research Vignette
3.3
Qualtrics Essentials
3.4
Qual-TRIX
3.5
Even moRe, particularly relevant to iRb
3.6
intRavenous Qualtrics
3.6.1
The Codebook
3.6.2
Using data from an exported Qualtrics .csv file
3.6.3
Tweaking Data Format
3.7
Practice Problems
4
Psychometric Validity: Basic Concepts
4.1
Navigating this Lesson
4.1.1
Learning Objectives
4.1.2
Planning for Practice
4.1.3
Readings & Resources
4.1.4
Packages
4.2
Research Vignette
4.3
Fundamentals of Validity
4.4
Validity Criteria
4.4.1
Content Validity
4.4.2
Face Validity: The “Un”validity
4.4.3
Criterion-Related Validity
4.4.4
Construct Validity
4.4.5
Internal Consistency
4.4.6
Structural Validity
4.4.7
Experimental Interventions
4.4.8
Convergent and Discriminant Validity
4.4.9
Incremental Validity
4.4.10
Considering the Individual and Social Consequences of Testing
4.5
Factors Affecting Validity Coefficients
4.6
Practice Problems
4.6.1
Problem #1: Play around with this simulation.
4.6.2
Problem #2: Conduct the reliability analysis selecting different variables.
4.6.3
Problem #3: Try something entirely new.
4.6.4
Grading Rubric
4.7
Homeworked Example
4.7.1
Check and, if needed, format data
4.7.2
Create a correlation matrix that includes the instrument-of-interest and the variables that will have varying degrees of relation
4.7.3
With convergent and discriminant validity in mind, interpret the validity coefficients; this should include an assessment about whether the correlation coefficients (at least two different pairings) are statistically significantly different from each other.
4.7.4
With at least three variables, evaluate the degree to which the instrument demonstrates incremental validity (this should involve two regression equations and their statistical comparison)
5
Reliability
5.1
Navigating this Lesson
5.1.1
Learning Objectives
5.1.2
Planning for Practice
5.1.3
Readings & Resources
5.1.4
Packages
5.2
Defining Reliability
5.2.1
Begins with Classical Test Theory (CTT)
5.2.2
Why are we concerned with reliability? Error!
5.2.3
The Reliability Coefficient
5.3
Research Vignette
5.4
A Parade of Reliability Coefficients
5.4.1
Reliability Options for a Single Administration
5.4.2
Reliability Options for Two or more Administrations
5.4.3
Interrater Reliability
5.5
What do we do with these coefficients?
5.5.1
Corrections for attenuation
5.5.2
Predicting true scores (and their CIs)
5.5.3
How do I keep it all straight?
5.6
Practice Problems
5.6.1
Problem #1: Play around with this simulation.
5.6.2
Problem #2: Use the data from the live ReCentering Psych Stats survey.
5.6.3
Problem #3: Try something entirely new.
5.6.4
Grading Rubric
5.7
Homeworked Example
5.7.1
Check and, if needed, format the data
5.7.2
Calculate and report the alpha coefficient for a total scale score and subscales (if the scale has them)
5.7.3
Subscale alphas
5.7.4
Calculate and report ωt and ωh
5.7.5
With these two determine what proportion of the variance is due to all the factors, error, and g.
5.7.6
Calculate total and subscale scores.
5.7.7
Describe other reliability estimates that would be appropriate for the measure you are evaluating.
6
Item Analysis for Educational Achievement Tests (Exams)
6.1
Navigating this Lesson
6.1.1
Learning Objectives
6.1.2
Planning for Practice
6.1.3
Readings & Resources
6.1.4
Packages
6.2
Research Vignette
6.3
Item Analysis in the Educational/Achievement Context
6.3.1
And now a quiz! Please take it.
6.4
Item Difficulty
6.4.1
Percent passing
6.4.2
Several factors prevent .50 from being the ideal difficulty level
6.5
Item Discrimination
6.5.1
Index of Discrimination
6.5.2
Application of Item Difficulty and Discrimination
6.6
In the
psych
Package
6.6.1
A Mini-Introduction to IRT
6.7
Closing Thoughts on Developing Measures in the Education/Achievement Context
6.8
Practice Problems
7
Item Analysis for Likert Type Scale Construction
7.1
Navigating this Lesson
7.1.1
Learning Objectives
7.1.2
Planning for Practice
7.1.3
Readings & Resources
7.1.4
Packages
7.2
Introducing Item Analysis for Survey Development
7.2.1
Workflow for Item Analysis
7.3
Research Vignette
7.4
Step I: Corrected item-total correlations
7.4.1
Data Prep
7.4.2
Calculating Item-Total Correlation Coefficients
7.5
Step II: Correlating Items with Other Scale Totals
7.6
Step III: Interpreting and Writing up the Results
7.7
A Conversation with Dr. Szymanski
7.8
Practice Problems
7.8.1
Problem #1: Play around with this simulation.
7.8.2
Problem #2: Use raw data from the ReCentering Psych Stats survey on Qualtrics.
7.8.3
Problem #3: Try something entirely new.
7.8.4
Grading Rubric
7.9
Bonus Reel:
7.10
Homeworked Example
7.10.1
Check and, if needed, format and score data
7.10.2
Report alpha coefficients and average inter-item correlations for the total and subscales
7.10.3
Produce and interpret corrected item-total correlations for total and subscales, separately
7.10.4
Produce and interpret correlations between the individual items of a given subscale and the subscale scores of all other subscales
7.10.5
Traditional Pedagogy Items
7.10.6
APA style results section with table
7.10.7
Explanation to grader
EXPLORATORY
FACTOR
ANALYSIS
8
Principal Components Analysis
8.1
Navigating this Lesson
8.1.1
Learning Objectives
8.1.2
Planning for Practice
8.1.3
Readings & Resources
8.1.4
Packages
8.2
Exploratory Principal Components Analysis
8.2.1
Some Framing Ideas (in very lay terms)
8.3
PCA Workflow
8.4
Research Vignette
8.5
Working the Vignette
8.5.1
Three Diagnostic Tests to Evaluate the Appropriateness of the Data for Component-or-Factor Analysis
8.5.2
Principal Components Analysis
8.5.3
Specifying the Number of Components
8.5.4
Component Rotation
8.5.5
Component Scores
8.6
APA Style Results
8.7
Back to the FutuRe: The relationship between PCA and item analysis
8.7.1
Calculating and Extracting Item-Total Correlation Coefficients
8.8
Practice Problems
8.8.1
Problem #1: Play around with this simulation.
8.8.2
Problem #2: Conduct a PCA with another simulated set of data in the OER.
8.8.3
Problem #3: Try something entirely new.
8.8.4
Grading Rubric
8.9
Homeworked Example
8.9.1
Check and, if needed, format data
8.9.2
Conduct and interpret the three diagnostic tests to determine if PCA is appropriate as an analysis (KMO, Bartlett’s, determinant)
8.9.3
Determine how many components to extract (e.g., scree plot, eigenvalues, theory)
8.9.4
Conduct an orthogonal extraction and rotation with a minimum of two different factor extractions
8.9.5
Conduct an oblique extraction and rotation with a minimum of two different factor extractions
8.9.6
Determine which factor solution (e.g., orthogonal or oblique; which number of factors) you will suggest
8.9.7
APA style results section with table and figure of one of the solutions
8.9.8
Explanation to grader
9
Principal Axis Factoring
9.1
Navigating this Lesson
9.1.1
Learning Objectives
9.1.2
Planning for Practice
9.1.3
Readings & Resources
9.1.4
Packages
9.2
Exploratory Factor Analysis (with a quick contrast to PCA)
9.3
PAF Workflow
9.4
Research Vignette
9.5
Working the Vignette
9.5.1
Data Prep
9.5.2
Principal Axis Factoring (PAF)
9.5.3
Factor Rotation
9.5.4
Factor Scores
9.6
APA Style Results
9.6.1
Comparing FA and PCA
9.7
Going Back to the Future: What, then, is Omega?
9.8
Comparing PFA to Item Analysis and PCA
9.9
Practice Problems
9.9.1
Problem #1: Play around with this simulation.
9.9.2
Problem #2: Conduct a PCA with the Szymanski and Bissonette
(2020)
research vignette that was used in prior lessons.
9.9.3
Problem #3: Try something entirely new.
9.9.4
Grading Rubric
9.10
Homeworked Example
9.10.1
Check and, if needed, format data
9.10.2
Conduct and interpret the three diagnostic tests to determine if PCA is appropriate as an analysis (KMO, Bartlett’s, determinant)
9.10.3
Determine how many components to extract (e.g., scree plot, eigenvalues, theory)
9.10.4
Conduct an orthogonal extraction and rotation with a minimum of two different factor extractions
9.10.5
Conduct an oblique extraction and rotation with a minimum of two different factor extractions
9.10.6
Determine which factor solution (e.g., orthogonal or oblique; which number of factors) you will suggest
9.10.7
APA style results section with table and figure of one of the solutions
9.10.8
Explanation to grader
Confirmatory Factor Analysis
10
CFA: First Order Models
10.1
Navigating this Lesson
10.1.1
Learning Objectives
10.1.2
Planning for Practice
10.1.3
Readings & Resources
10.1.4
Packages
10.2
Two Broad Categories of Factor Analysis: Exploratory and Confirmatory
10.2.1
Common to Both Exploratory and Confirmatory Approaches
10.2.2
Differences between EFA and CFA
10.2.3
On the relationship between EFA and CFA
10.3
Exploring a Standard CFA Model
10.3.1
Model Identification for CFA
10.3.2
Selecting Indicators/Items for a Reflective Measurement
10.4
CFA Workflow
10.4.1
CFA in
lavaan
Requires Fluency with the Syntax
10.4.2
Differing Factor Structures
10.5
Research Vignette
10.5.1
Modeling the GRMSAAW as Unidimensional
10.5.2
Modeling the GRMSAAW as a First-Order, 4-factor model
10.6
Model Comparison
10.7
A concluding thought
10.8
Practice Problems
10.8.1
Problem #1: Play around with this simulation.
10.8.2
Problem #2: Use simulated data from other lessons.
10.8.3
Problem #3: Try something entirely new.
REFERENCES
Published with bookdown
ReCentering Psych Stats: Psychometrics
Confirmatory Factor Analysis