Dear Doctoral Colleagues,
In the following series of short lecturettes, I outline the elements of what might be called an “ideal” manuscript. The elements are as follows:
As you all progress in doctoral studies and reading journal articles, you undoubtably have come to the realization that all empirical journal articles follow very similar format. And, this format mirrors the five chapters of a dissertation. The major difference, of course, is the length. A typical journal article is between 15 to 25 pages while a dissertation can go for up to 300 pages with appendices and references.
The purpose of these lecturettes is to help you (1) critically read journal articles (and, thus, write annotated bibliographies and even a peer-reviews;(2) assist you in learning how to write a stronger dissertation, and (3) know the elements of the major sections of the research article/dissertation.
Depending upon where you are in your doctoral studies, these lecturettes will have a different meaning to you. If you are at the beginning of your studies and have not yet conducted a research study of your own, then the purpose of the lecturettes is primarily as a vocabulary lesson, one you should memorize. That is, terms such as conceptual framework will be new to you as will be the notion of methods and research designs. Each of these lecturettes is written in scholarly terms that are unfamiliar to general readers, and so you may be reading them for basic comprehension. If you are more advanced in your doctoral studies and are already a novice educational researcher, then these lecturettes are sort of a “how-to” manual for reminding you of best practices in APA writing. In other words, the lecturettes can be used again and again throughout your doctoral journey.
What separates published scholarly work from drafts is something called “peer-review,” which is a matter of close analysis and critique. Most academic journals require that manuscripts be peer reviewed by two or more “blind” readers. These readers are typically given two rubric formats: first, a numerical rating scale by categories (e.g., quality of writing, methods, relevant literature, etc.), and secondly, a blank space for writing constructive criticisms (feedback) to authors on how to improve the manuscript. Even manuscripts that are rejected from journals will receive constructive criticisms so that the authors may benefit from their submission to that journal. Specifically, manuscript review forms also ask reviewers to rate the significance of the study, its conceptual framework, the potential contribution it makes to the literature, the replicability and appropriateness of the methods used, the writing style, and the validity/trustworthiness (or generalizability) of the findings.
The purpose of these lecturettes is to guide you through the steps of writing and peer reviewing so that you can both improve your own manuscripts and also make good judgments about the quality of journal articles themselves.
Lecturette 1: Theoretical or Conceptual Frameworks –
The key word for you to see is “or.” Yes, a conceptual framework can be labeled a theoretical framework and, therefore used synonymously. But if we want to be precise, a theoretical framework is based on theories, models and paradigms that are already in the literature. In contrast, a conceptual framework highlights key ideas (aka concepts) and their relationships to other concepts. Identifying your framework is a critical and necessary component of the research effort.
EVERY research article has either a theoretical or a conceptual framework. Unfortunately, not all authors make her/his/their framework explicit. We -the reader- should not have to guess or intuit which theories or concepts are involved. This is the researchers/authors responsibility. Therefore, the first task of an author/researcher or reviewer is to ensure that either a theoretical or a conceptual framework is stated explicitly. Ask yourself: Does the author connect her/his/their ideas to other concepts/ideas/theories in the literature? Does the author acknowledge her/his/their debt to the ideas of others? Does the author recognize the paradigms, politics, and/or subjectivity of her/his/their writing? To me, an author's acknowledgement of the framework is the hallmark of intellectual honesty. It is intellectual because it acknowledges the ideas of others; it is honest in that it has been made explicit rather than hidden between the lines. But be wary/skeptical/doubtful. Just because the author has identified a framework, e.g., democracy, Marxism, postmodernism, behaviorism, rationalism, etc., it does not mean that this statement by itself is sufficient. Ideas/concepts/theories need to be developed through breadth and depth of analyses and syntheses.
In other words, don't let a sentence or a short paragraph suffice. Thus, as a reader or reviewer, your task is to judge the development of a theoretical or conceptual framework. A good exercise is to create an APA table of the theories/concepts being cited and then trace the evolution of each across time (e.g., chronology) or space (e.g., global, international, local). In so doing, the author communicates to the reader how the ideas/concepts/theories have changed over time and just as importantly why you or the author has settled upon one particular meaning/definition of a word or concept. Keeping a Glossary of you key terms with citations is very helpful.
Once you decide whether or not the author has acknowledged her/his/their conceptual framework, and once you decide that it has been developed sufficiently, then there is a third significant decision you need to make. That is, how well/completely does the author make connections across the different concepts/ideas/theories being used? This third objective has to do with the logic of the author's argument. This is a quality judgment on your part: how well (i.e., explicitly) does the author make connections across ideas? This criterion is not the same as writing style. I think we would all acknowledge that educational research is not easy bedtime reading. It is often dense, complex, intricate, etc. Once you get past the difficult language usage, however, it still has to make sense. If it does, then good. If not, then help the author [and yourself] clarify the logical connections. If it’s your own work that we are talking about, then ask a critical friend to look at your logic. I often ask my doctoral students to draw an APA Figure with arrows connecting their ideas/concepts/theories. Just the very act of drawing an APA Figure can help you to visualize and test the logic of the framework.
Lecturette #2: Review of Literature
The review of literature is closely related to the conceptual framework discussed previously. Some authors choose to create two separate sections: one for their theoretical/conceptual framework; another for their review of related literatures. Whether separated or integrated, both sections must be identifiable, explicit, and well developed. It is essential, however, that the researcher be aware of the relationships among the different authors being cited. I often refer to the groupings of researchers as “families.” Knowing the family of authors/researchers helps you to see a paradigm or a tradition or a philosophy – aspects that connect one researcher to another. A review of literature, therefore, is more than just citing random authors who seem to agree with one another, As a writer or a reviewer, you need to see the chronology of ideas moving from seminal authors to related authors over time and across space (geography). A strong review of literature will cite all of the seminal research/theory on the topic. For a reviewer who may not be thoroughly familiar with a topic, we have to trust the author with respect to seminal authors and the logic of the argument. Nevertheless, as a reviewer, you/we can make judgments on the (1) organization, (2) timeliness, and the (3) depth of the review. The same is true for you as an author.
Before explaining one through three, I want to talk about a practice that upsets me when reading some reviews of literature. I often read sentences followed by a long list of names and dates, supposedly of books and articles that support [or not] the author's position. And that's it. No explanation, no discussion of the citations, no mention of research methods, just names and dates. What about the research methods used by the authors? What about the sample/participant size? What about how data were collected? Just listing authors is not good research writing, and you won't see that kind of writing in published reviews in the journal titled, REVIEWS OF EDUCATIONAL RESEARCH and other top-tier journals that publish complete/systematic reviews of literature. You will, however, see this sloppiness in too many published open access educational research journal articles. Please don't do this in your own writing. If you are going to cite an author, share the reason why. Tell the reader about the author's findings, methods, and context. Then trust that the reader will make a decision on the validity of the citation. If one purpose of the review of literature is to establish your connection to the field, then make the connection, limitations and all. I will return to this point under the topic below labeled "depth."
A good review of literature should be well-organized, timely, and provide enough background or depth so that the reader can understand the research study of the manuscript.
1. Organization: It is the author's responsibility to decide what to include in a review of literature and how to organize the concepts into subheadings. You can't include everything in a review nor can you present it as one long run-on sentence. Criterion one: does the review offer a cohesive argument? Criterion two: does the author break down the argument into digestible parts, with sub-headings? Criterion three: does the review lead to specific the research questions that ask about the gaps in the literature? Exception: in a grounded theory study, much of the literature will appear as part of the findings and discussions nevertheless, the organization, timeliness, and depth will still have to be there. If the above criteria are present, then the review is well-organized.
2. Timeliness: Many authors ask how far back should one look in conducting a review of literature? My answer is that you go as far back as the origin of the concepts/theories/ideas. I am not a fan of impact factor formulae that give the greatest weight to the most recent citations. In education, good ideas remain for decades, if not for a millennium. Yet, too many research articles and authors seem to think that only the most recent research studies are worthy of citations. Yes, a review must be timely and up-to-date. If the study is contributing to existing theory/knowledge, then you can't leave out the recent works. At the same time, some of the earliest works which shaped the field/topic are also relevant and should be cited. I like to see a balance of old and new. This will add to your credibility as a scholarly author.
3. Depth: Let's go back to my pet peeve of having a long list of names and dates in parenthesis at the end of a sentence. That's okay as a topic sentence of a section or paragraph. It's not okay as a stand-alone assertion. What I need to see is how the citations support the ideas of the argument being made. Not only do I want to see what the cited work actually said, but I also want to know how the research was designed and carried out. Not all citations carry equal weight. Not all studies have the same validity, reliability, trustworthiness or believability. It's the author's responsibility to make professional judgments about the cited works and to share those judgments with the reader. A review of literature is an original scholarly work precisely for the reason that each one of us makes different decisions with respect to the above subtopics. Given the proliferation and debates surrounding valid and reliable knowledge, the tone of the review should reflect a degree of humility. We can express this by using words such as "caution," "tentative," etc. What the author writes is only part of the contribution to knowledge. Readers have a responsibility to make good decisions, too. I am not now talking about the conceptual framework labeled constructivism; I am talking about writing and reading as a shared intellectual process that demands openness and honesty. Just a decade or two ago, there was a debate over methods, quantitative versus qualitative. That debate enriched our field. I think we need more open, scholarly debates on the purpose and presentation of reviews of literature. Today, there is no one "model" of a review of literature.
In the next lecturette, number 3, we'll talk about methods, which all have limitations and delimitations. An analogous situation exists with reviews of literature. It is intellectually honest to report on studies that present evidence contrary to a review's arguments. Yet, this practice is becoming a rarity. As a peer-reviewer, I would give higher scores to a manuscript that included disconfirming evidence from other research studies. I want to have confidence in an author: honesty and validity give me that confidence. Finally, there are two additional writing practices that are, for me, questionable: (1) authors who cite their own work unjustifiably; and (2) authors who cite sources from a canon of acceptable works. As for point one, it is possible to find yourself reading a manuscript by a theorist who has written all of the seminal works on a topic. In that case, self-citing is indeed justified. As for the canon of citations, it is important to acknowledge those dominant discourses that define any field; it is also important as a researcher to seek out authors whose works have challenged the dominant discourses. This is a mark of a thorough – and honest - researcher.
Lecturette #3: Methods
From the two previous lecturettes, you know that I have been discussing methods as part of both the THEORETICAL/CONCEPTUAL FRAMEWORKS and REVIEWS OF LITERATURE. In other words, I do not believe that the separation between METHODS and CONCEPTS/THEORIES/LITERATURE serves the purposes of social science/education research. All too often, we use the terms "what works" and "best practices" interchangeably with "scientific evidence" and "research-based." As a result, neither policymakers nor the lay public can distinguish among these terms, and, thus, their usages of the terms are not consistent with scholarly discussions, but rather fuel the ideological and political debates surrounding education. Practitioners and many professors find themselves caught up inside of the debates, but often lack the knowledge and/or power to influence them. The separation of methods and concepts/theories/literature promotes the use of educational discourse without understanding the claims for validity, reliability, and/or trustworthiness.
If we take the concept of leadership as an example, we find that it had been first conceptualized as a uni-dimensional concept, then as a two-dimensional concept, then as a multi-dimensional concept, and then, perhaps, as a concept contingent upon contexts. It may emphasize traits, tasks, and situations, styles, strategies, morality, a contested concepts, etc. etc. Some of these conceptualizations were based on descriptive survey research; others were based on interview data and analysis. In some studies, the samples were military officers; in other studies, the participants were CEOs or accountants or boyscout leaders. Tell me how valid and reliable the findings of these studies are when applied to school teachers, professors, principals, university presidents and district superintendents!? What's at stake here is the integrity of our knowledge-bases. It is up to the scholarly community to debate the meanings of "research-based" and to educate policy makers, practitioners, and the public of its various meanings and limitations.
To allow others to tout their favorite programs as "best practices" or "what works" without presenting the evidence is irresponsible. Methods are how we establish findings and conclusions as valid, reliable, and/or trustworthy. In the various journals for which I write and review manuscripts, the methods' sections vary from one page up to five pages. Regardless of the space allotted to methods by journals, the researchers need to present a discussion of the research design, then move to the sample [i.e., who and how they have been selected], the data [i.e., variables of interest], methods of data collection-including instruments, and how the data will be SYSTEMATICALLY analyzed. Every research study has a method, that is, a process by which the author has moved from her/his assumptions and assertions questions, answers and to conclusions.
In research, it is important to be explicit about which methods are being used and how the study itself was conducted [i.e., procedures]. As best as I can, I will make no distinctions between quantitative and qualitative methods. To me, the research question and research design influence which methods will be most suitable for carrying out the study. In other words, a researcher is a researcher regardless of the method used. Your task as an author/reader/reviewer is to determine whether the author has (1) given a justification for why she/he chose a particular method for the study, and (2) provided readers with enough detailed information so that another researcher could replicate the study's procedures and analyses.
1. Justification: Sometimes, the theoretical or conceptual framework will influence [or dictate] which method should be used. A feminist researcher might argue in favor of elevating personal voices in order to empower the participants of the study; a linguist might take a structural, grammatical, poststructural, conversational or critical discourse approach. In each instance, the theories and conceptual framework itself are influencing the research questions which then influence the methods. Another justification regarding the choice of method might be based on a review of literature. For example, if the topic has been studied thoroughly from one particular paradigm, an author might ask, would the results be the same or different if another paradigm was to be used. We see this in the quantitative-qualitative choices. This decision is like putting on a different pair of glasses and looking at a phenomenon again, but this time from a different lens. Whatever the argument, you should look for its justification in the manuscript. If it is missing, then that's a serious weakness.
2. Detailed Procedures: Here I become very simplistic. I want to see the same kind of step by step writing that is found in good cookbooks,-the ones we actually use in our kitchens. I don't want anything left out. What did the author do first, second, third, etc.? Tell me everything that was done. The more research you read, the more you can distinguish between a quality study [valid, reliable and well-written] and one that falls short for some methodological reason. Many of us prefer to review articles using a particular method because we are unfamiliar with the intricacies of a method. My advice is to learn as much about all research methods as you possibly can. In so doing, you discover that every method and every design has some limitations and delimitations. I give high marks to those authors who tell the reader what those limitations [of their methods and sampling] and delimitations [of context and setting] are.
Lecturette #4: Findings
The fourth and final lecturette is on reporting results. Manuscripts use different terms in this section, such as findings, analysis, discussion, conclusion, summary, etc. There are a number of factors to look for in these sections: (1) does the author answer the research question[s]? (2) does the author go beyond the data and offer her/his/their interpretations? (3) do the findings make a contribution to existing knowledge? and, (4) does the author acknowledge the limitations of the study?
Answering the research questions: (1) A strong methods' section will tell you not only how the author intends to answer the research question, but will also give you confidence that the reported findings indeed emerged from the data themselves. It is important that the findings be tied directly to the data. At the same time, it is important to differentiate among findings, answers to a research question, and answers to specific questions on the instruments [e.g., survey, interview protocol, etc.]. While it is possible for an item on a research instrument to correspond to a research question, that is not likely to be the case. Typically, the authors have to analyze and synthesize the data from item responses in order to present an answer to the research question. Of course, the research question/s was asked before the study was conducted; therefore, the researcher may have findings that emerged which are not answers to the research question. The point is that a researcher learns new knowledge from doing the study. Reporting findings and answers to research questions require analysis, synthesis, and interpretation.
An answer to an instrument's question is simply reporting responses from the instrument. However, writing up the findings are more complicated. All of the findings/answers [i.e., research and survey/interview questions] are inextricably connected. Thus, an important distinction to make is between raw data and findings. The raw data allow the author to analyze, synthesize and interpret. Both raw data and research findings need to be in the FINDINGS section. Here again, experience will tell you when the author has achieved a good balance. Too much raw data and the reader can get lost in details and trivia. Conversely, a rush to a conclusion-without providing the systematic steps of analysis-can lead the reader to question the validity of the study itself. In reviewing [and in you own writing], look for sufficient data to justify what the author is claiming to be a finding.
Meaning and interpretations: (2) The analysis, synthesis, and interpretation should all add up and make sense. While the reader has a responsibility to find meaning in the study, the first responsibility falls to the author to say what she/he found and what it means. If you as the reader disagree, okay. But as a reviewer, your task here is to determine whether the author is justified in coming to her/his conclusions.
All of us have our own opinions. Yet, the reason we conducted the study in the first place was to learn what others see, think, believe, or feel. As you write up the findings, make sure you yourself acknowledge whether the meanings come from your participants/subjects or from your interpretations. This is not an easy task, but it marks you as an ethical researcher.
Contributions: (3) It is important that the researcher return to her/his/their theoretical or conceptual framework and to the review of literature. If the findings confirm existing knowledge, then the author should say so and link it to past studies. If, however, the findings reveal a new direction, a new variable of interest, or something that is original, then that becomes the findings [and your contribution to the literature]. Either way, it is important for the author to tie the study's findings back to existing knowledge and theories.
Limitations: (4) The findings represent findings from this one study alone. Intellectual honesty should guide the author in not overstating the results or implications of the study. Hopefully, the research question[s] has been answered and what it all means has been stated clearly. Beyond that, findings should be reported with care and humility.
Abstract
I am putting this discussion of abstracts at the end. Why? Because you write the abstract after the study has been completed. Yes, it then goes upfront as the roadmap, but these lecturettes are for teaching.
Begin your abstract with Background/Context. Keep this sentence or two or three general. Then, go to Purpose/Objective/Research Questions. Here is where you are specific and direct. Setting is important especially if it is different from the context. Tell your readers who were your participants, how many, and give whatever other demographic is relevant to the study. The Research Design jumps to your methods and your methods jump to Data Collection and Analysis. Be specific, sequential and succinct. Next, two or three sentences on Findings/Results, followed by Conclusions/Implications/Recommendations – whichever seem most appropriate to the study.
###
Much more can be said about the “ideal” manuscript. My purpose here in the lecturettes was limited to helping readers and writers internalize the genre of journal writing. Scholarship, of course, goes beyond journals and dissertations. To the extent that such writing contributes to the knowledge base, I find the ideas presented here to be basic and necessary. Beyond that, it is our responsibility to advance our quality of writing and to take our scholarship in new and significant directions.