EDUC 607 - Murillo
Text:
Locke, L. F., Silverman, S. J., & Spirduso, W.W. (2010). Reading and Understanding Research. (3rd ed.) Thousand Oaks, CA: Sage Publications.
Research report evaluations are due in-class session 6 (20 points)!! You have to select, read and evaluate one Quantitative research report (10 points) and one Qualitative research report (10 points).
Steps: 1) Select 2) Read 3) Evaluate
Step 1: Use the following questions as a guide when selecting credible research reports.
1) Has the paper been peer reviewed for a refereed journal?
2) Is evidence of replication available to support the results?
3) Is a conflict of interest evident for the person/s) doing, sponsoring, or disseminating the study?
4) Can the question(s) asked be answered in the study?
5) Is evidence of technical problems apparent in design or analysis?
6) Are sample composition and size adequate to address the question(s) asked and to support the conclusions reached?
7) Are the conclusions offered supported by the findings?
8) Is there indication that the investigator was careless in conducting or reporting the study?
9) Does the author say things about the study that appear to be examples of a poor understanding of scholarship?
10) Is the author conscientious in frankly drawing your attention to limitations imposed by the design or sample, or compromises made to circumvent problems?
11) Did you encounter any other reasons for suspending trust in the study?
12) Do you understand all of the report, or, in all honesty, do you require assistance with some elements?
Step 2: The basic problem in reading research is that: “Researchers write for other researchers (or, at least, for people who are insiders in the area of inquiry), and they have little motive to make themselves understood by outsiders. The reports they write are the primary (though not the only) vehicle for a continuing conversation among active scholars in an area of investigation”. Therefore keep the following tips in mind for reading research:
- Begin with a basic attitude of respect for the researchers.
- Understand necessary compromises between “perfection and the practicalities of time and money.”
- Reading a research report is like studying - underline, highlight, flip pages, make notes, draw diagrams, etc. You will rarely read a report start to finish in a linear way.
- Make a map or flowchart of the major temporal steps of the study. This helps to visually represent what occurred and to cut down on confusion about process.
- Don't get stuck on unfamiliar words. Try and understand them in context, or write them down for later.
- Don’t get stuck on what the author didn’t report. This will be common, as you will look for information that is just not reported. Make a note about what is missing and move on.
- Don't get stuck on statistics. Rule of thumb: “if the technique is unfamiliar, look in the text and not the table” for a description. Worse case scenario, just skip them. “Don't let skipping something like statistics panic you, and certainly do not let it make you feel guilty or inadequate. Statistics have a practical purpose, but they are not magical incantations that hold mysterious power. They are just tools.”
- Don't get stuck on the question of whether or not the research is good.
- Quality in research is always a matter of degree, and - there is never a perfect study. Wait to make a judgement until you have read the whole report.
- Harbor respectful skepticism. Total columns, check citations, see if sample size remains the same, are all research questions answered?
Step 3: Once you have selected and read one credible Quantitative research report, and one credible Qualitative research report (using the above criteria), use the following formats to evaluate each!!!
*** Use these questions for your written paper, which you turn in to me:
Quantitative = “12 Steps to Understanding a Quantitative Research Report”
and
Qualitative = “12 Steps to Understanding a Qualitative Research Report”
Quantitative
1) What study report is this? (Record a full reference citation.)
2) What kind of study is this?
3) What was the general purpose of the study? What questions does it raise?
4) How does answering the research question(s) add something new to what is already known? If the study is a replication, why is that important?
5) Who or what was studied? (number and key characteristics)
6) In sequential order, what were the major steps in performing the study? (Record these in a flowchart) Do not just repeat details from Items 1-5 and 7-10. Create an explanatory sketch that a year from now would help you recall how the study was done.
7) What data were recorded and used for analysis? (e.g., questionnaire responses, test scores, fieldnotes, meter readings, etc.)
8) What kind(s) of data analysis was used? (e.g., statistical, logical categorization, etc.)
9) What were the results? (After analysis, what do the data from Item 7 say about the question(s) raised in Item 3?)
10) What does the author conclude? (In light of both Item 9 and the entire study experience, what is said about Item 3?)
11) What cautions does the author raise about interpreting the study, and what do you think are important reservations?
12) What particularly interesting or valuable things did you learn from reading the report? (Consider results, method, discussion, references, etc.)
Qualitative
1) What study report is this? (Record a full reference citation.)
2)Who is the investigator? Include personal history, particularly as related to the purpose, participants, or site of the study.
3) If made explicit, what type of qualitative research is this? Is the author working from a feminist, Marxist, interpretivist, symbolic interactionist, critical theorist, or other vantage point?
4) What is the purpose of the study? What are the focusing questions (if any)? Is the purpose primarily theoretical, practical, or personal?
5) Where does the study take place, and who are the participants? Describe the general physical and social context of the setting and salient characteristics of the main actors. If this is not a field study, describe the setting and participants presented in the secondary data source.
6) In what sequence did the major elements of the study occur? Describe (or diagram in graphic format, such as a flowchart) timing, frequency, order, and relationships used in organizing the study.
7) How were data collected? Was recording done through observation and fieldnotes, taped interviews with transcription, document analysis with record forms, or some combination?
8) If this was a field study, what was the author’s role while collecting data?
9) What procedures were used for analysis of data? Was constant comparison used, were categories developed inductively, were themes constructed, was computer software employed?
10) What were the results? In general terms, what is the answer to the question, “What was going on there?”
11) How are design or research methods used to enhance the credibility (trustworthiness and believability) of the study?
12) What parts of the study did you find powerful or particularly instructive? What was moving or striking, and what provided new insight?