Assignment Research paper on How to evaluate scientific reliability and validity

Assignment #7: How to evaluate scientific reliability and validity
Must be turned in before: July 3, 2020 before 11:59 PM.
READING
a. To learn what reliability and validity mean in research:
1. Read an excerpt from Sommer’s (no date) article, “Introduction:
Reliability and Validity.”
2. Read a student from Bangor University’s (2011) blog post, “Reliability vs.
Validity.”
b. To learn what external versus internal validity means in research, read Bradley’s
(no date), PPT, “Reliability, Validity, and Bias.”
1. Although you should be sure you understand both terms, reliability and
validity, and their difference, this assignment will focus on reliability.
(Another assignment will focus on validity.)
c. To dig even deeper into the issues of reliability, as well as replicability, and
reproducibility, read Munafó et al.’s (2017) article, “A Manifesto for
Reproducible Science.”
1. NOTE: Munafó et al.’s (2017) article is complex (it’s not Huffington Poststyle writing).
▪ You might need to read it a few times to understand it. Please be
persistent.
2. Munafó et al. (2017) suggest 10 procedures for making science, including
psychological science, more reproducible (replicable and reliable).
▪ Their 10 suggested procedures are listed in Table 1 on the third
page, and each is explained in the article.
WRITING ASSIGNMENT
3. Choose three of Munafó et al.’s (2017) 10 suggested procedures.
▪ Choose the three suggested procedures that you like the most (or
that you understand the best).
4. Write one five-paragraph Examples-Style essay to support the thesis
“Scientists Can Improve Research Reliability.”
▪ For this essay, the three procedures you have chosen from Munafó
et al.’s (2017) article will be your three ‘examples’ in your
Examples-Style essay.
d. If you need to review how to write a Five-Paragraph Essay, including how to write
an Examples-Style essay, watch again the lecture video, “The Five-Paragraph
Model” (a transcript of the video is available here).
Part 1: Reliability and Validity (primer)
1. Check your essay to make sure your Introduction Paragraph has a hook
and a Thesis Statement.
2. Check your Thesis Statement to make sure that it summarizes your three
examples/procedures.
3. Check your essay to make sure it has three Supporting Paragraphs.
4. Check each of your three Supporting Paragraphs to make sure each one
has a Topic Sentence, three Supporting Sentences, and a Conclusion
Sentence.
5. Check your essay to make sure it has a Conclusion Paragraph.
6. Check your Conclusion Paragraph to make sure it has a sentence that
restates your Thesis Statement (summarizing your three
examples/procedures).
e. Because you’ll be drawing a lot of ideas from Munafó et al.’s (2017) article, make
sure you don’t plagiarize.
1. Be sure to include in-text citations to Munafó et al.’s (2017) article when
needed, but try to not make Munafó and his co-authors the subjects of
your sentences.
2. Make your ideas (e.g., the three procedures you’ve chosen) the subjects
of your sentences.
3. Remember, from Assignment #5a, how to synthesize psychologically
science, not Mad Lib it.
4. Include a full citation to Munafó et al.’s (2017) article, in APA style, at the
end of your essay.
5. Save your essay as a PDF and name the
file YourLastname_ReliabilityEssay.pdf.
f. Go to “Course Materials” → “Assignment #7: How to evaluate scientific reliability
and validity” and attach your essay, saved as a PDF.
READING
a. Reliable and valid studies have reliable and valid operational definitions. To learn
what operational definitions are, read Bluestone’s (no date) PPT slide,
Operational Definitions
b. To understand why consistent, precise, logical, and justifiable operational
definitions are crucial for reliability and validity, read Dewey’s (2007) chapter,
“Operational Definitions.”
c. To appreciate the difficulty of generating reliable, precise, and valid operational
definitions, read this excerpt from Lammers and Badia’s (2013)
textbook, Fundamentals of Behavioral Research.
Part 2: Operational Definitions
WRITING ASSIGNMENT
d. To demonstrate the difficulty of generating reliable, precise, and valid operational
definitions, complete this fillable PDF, which contains a list of 10 fortunes, such
as one would see in a Magic 8 Ball game. Each fortune has at least one term
bolded.
1. First, download the unfilled PDF and save it on your own computer.
▪ Rename the unfilled PDF to be YourLastName_PSY430_
OperationalDefinitions _Fillable.pdf. (In other words, add your
LastName to the beginning of the filename.)
▪ On your computer, open a PDF writer app, such as Preview, Adobe
Reader, or the like.
▪ Be sure to open your PDF writer app before you open the unfilled
PDF from your computer.
▪ From within your PDF writer app, open the unfilled PDF, which you
have already saved onto your computer and re-named.
2. Second, using your PDF writer app, fill in the PDF with your operational
definition of each of the bolded terms.
▪ For example, if the fortune is “You will inherit some money or a
small piece of land,” fill in the PDF with your operational definition
of “some money” (What are the criteria for “some money”? How
will you distinguish “some money” from money that’s not “some
money”?) and with your operational definition of “a small piece of
land” (What are the criteria for “a small piece of land”? How will you
distinguish “a small piece of land” from a large piece of land or no
land at all?).
▪ If you’re unclear what an operational definition is, it would be good
to go back and read the articles in a., b., and c., above.
3. Third, teach a friend (or family member) what operational definitions are.
▪ Then, ask this friend/family member to tell you their operational
definition of each of the bolded terms.
▪ Fill in your friend/family member’s operational definitions on your
PDF.
▪ Although you will first need to explain to your friend/family member
what an operational definition is, do not tell your friend/family
member your operational definitions of the bolded terms before
your friend/family member tells you their operational definitions
4. Fourth, teach a second friend (or family member) what operational
definitions are.
▪ Then, ask this second friend/family member to tell
you their operational definition of each of the bolded terms.
▪ Fill in your second friend/family member’s operational definitions on
your PDF.
▪ Again, you will need to explain to your friend/family member what
an operational definition is, BUT do not tell your second
friend/family member your operational definitions of the bolded
terms (or your first friend/family member’s operational definitions)
before your second friend/family member tells you their operational
definitions.
e. Go to the discussion board forum Assignment #7, Part 2: Operational Definitions
and make a new post of at least 200 words in which you do the following:
1. Identify which of the bolded terms you and your friends/family members
showed the most and the least reliability when each of you
(independently) defined those terms operationally.
▪ Speculate why those terms led to the most versus least reliability.
2. Identify which of the bolded terms you and your friends/family members
operationally defined in the most versus least valid ways.
▪ Speculate why those terms led to the most versus least validity.
3. Attach your filled-in PDF.
READING
a. Although “validity” is a general research methods term, when it is applied to tests,
such as IQ tests (or the SAT/ACT and the like) or personality inventories, we use
the specific term “test validity.”
b. To learn what the specific term “test validity” means and to be introduced to the
major ways that test validity can be assessed, read a section from Heffner’s (no
date) “Chapter 7.3 Test Validity.”
c. Now, think back to one of the exams you took last semester in one of your
courses. In fact, think about the most memorable exam you took (either because
it was memorably positive or memorably negative). Refer to this exam as “Text
X.”
WRITING ASSIGNMENT
d. Go to the discussion forum Assignment #7, Part 3: Different flavors of validity and
make a new post of at least 200 words in which you discuss Test X’s
1. construct validity (did the exam measure what it was supposed to
measure?);
2. concurrent validity (was the exam like other comparable exams, for
example, the exams your friends took in a similar course?);
3. content validity (were the items on the testing representative of the content
you were expected to learn?); and
4. predictive validity (how well did the exam predict your ability to use the
information in the future?