[pic]TOOL: Making Sure Data Are Valid and Reliable
Purpose: Before you can confidently interpret and analyze your evaluation data, you must ensure that the data you collect are valid and reliable. Otherwise, they won’t adequately support your outcomes. Use the questions and suggestions in this table to ensure the data you collect are valid and reliable.
Instructions: 1. Review the “Questions to Consider” in the table to ensure that the data you collect for your evaluation are valid and reliable, and that your sample size is adequate.
2. Consider the suggestions or proactive measures you might take to ensure ...view middle of the document...
|across studies. | | |
|Have we ensured that our measures are reliable? |Discuss with your evaluator how you will test for reliability of your instruments. You may| |
|A reliable measure produces stable responses |want to borrow from existing instruments, have an expert panel review and react to new | |
|regardless of the data collector. An unreliable |instruments, or pilot test the instruments in real settings and among members of your | |
|measure will yield varied responses depending on |target audience. Build in time and resources to test for reliability and calculate | |
|differences between interviewers or data |reliability coefficients so that you can assure stakeholders that you have strong | |
|collectors. |instruments. | |
| |Note that qualitative data collection (e.g., observations, open-ended interviews) poses | |
| |different validity challenges. Ideally, the instruments you create for these purposes will| |
| |require low levels of inference. Provide time and resources for researcher training in use| |
| |of the instruments to minimize differences in participant responses across data | |
| |collectors. | |
|Do we have the right data? |You will need an adequate sample size to ensure your data are valid. Make sure you begin | |