Critical Analysis of Roth and Bowen’s Paper: “When Are Graphs Worth Ten Thousand Words? An Expert-Expert Study”

Purpose and Question of the Research

The authors present the research purpose very explicitly in the paper. It can be found on the page 430, second paragraph, in which they state that: “this study was conducted to better understand graphing expertise. We were particularly interested in understanding the contributions of experience (content represented, laboratory experience, and understanding of conceptual frameworks) to the particular readings provided by scientists”. The research purpose is also written explicitly in the discussion (see p. 466, first line of the first paragraph): “… to better understand readings of familiar and unfamiliar graphs by professional scientists”.

The research question is not formulated explicitly in the paper. As the reader, I do not know exactly whether the authors forgot to mention the research question intentionally or not. But, the title of the paper is written in the form of a question, when are graphs worth ten thousand words. Later, in the conclusion (see p. 470, the last paragraph), the authors say that this title is their initial question, but not the research question. In my perspective, this initial question cannot be categorized as the research question or even as a good research question. Although it is researchable, this initial question does not provide any clear explanation about the things that the study wants to investigate. There is no any explanation in the question about the kinds of graphs that are going to be used. The word ‘when’ is also somewhat ambiguous. What does the term when means? Does it mean the time of presenting the graphs, or the kinds of graphs? But, this initial question is worthwhile to investigate. The researchers formulate the reason for it (see p. 430, paragraph one): “… there is little work on the actual use of graphs in everyday science, or on scientists’ reading of unfamiliar graphs”.

Research Approach/Strategy & Methods/Techniques

The study is a qualitative ethnography research. It is stated quite clearly in the paper (see p. 438, paragraph two): “… a situation often discussed as a danger in the literature on ethnography as method…”. The authors also show the research approach implicitly in the paper (see p. 437, paragraph three): “… we developed a successful methodology for analyzing graphs, graphing, and graph use from an anthropological perspective through successive methodological refinements culminating in the work presented here”. The research approach should fit the purpose of the research that the researchers are trying to achieve (Denscombe, 2010). According to Denscombe (2010), there are three key questions to decide whether the research approach is the best approach to achieve the goal, namely:

Is it suitable?

In my own perspective, the research approach, ethnography, is suitable for achieving the purposes of the research. It is because ethnography allows the researcher to have a holistic view of what the scientists think and interpret about graphs. It is also appropriate for the researchers to describe the scientists’ understanding of familiar and unfamiliar graphs.

Is it feasible?

Ethnography is a good research approach in relation to the access of the data sources because it employs an in-depth interview that can accommodate the scientists. The researchers did the interview either in the scientists’ offices or in the principal investigator’s office or laboratory (see p. 436, paragraph four). Although ethnography research needs a relatively long time (Denscombe, 2010), the researchers were able to achieve it since they had already started over the past 9 years (see p. 437).

Is it ethical?

The researchers have taken into account the importance of research ethics in their study. They use the code S01 – S16 (see p. 432) to indicate and describe the sixteen scientists in their study and to take the quotes from the interview transcripts (for example, see p. 444).

The researchers are quite open in describing the methods/techniques that they used in the study. According to the text, they used an interview method (see p. 436). The good side from the method of this study is that an interview is the most important data-gathering method in the ethnography studies. By doing interviews, the researchers are able to put what they have seen, heard or experienced into a larger context. Unfortunately, the researchers forget to specify the kinds of interview that they used. So, we cannot be really sure whether they used structured, semi-structured, unstructured, or retrospective interviews. However, in my perspectives, the text indicates that the researchers used unstructured interview in their study (see p. 437, paragraph two): “… scientists were asked to tell us as much as they could about the graph… they were asked to read aloud, when they were reading, and to describe what they were seeing when they looked at graph”. This paragraph also indicates that the researchers used open-ended questions in conducting the interview. This is also a good point for this study because an open-ended question allows participants to provide a long and detailed explanation about their interpretations of familiar and unfamiliar graphs.

Relation between theoretical background, question and methods

The researchers present the theoretical background of their study quite clearly. In the beginning, the researchers explain about the importance of graphs in the scientific practice (p.429). They also show typological and topological processes of interpreting graphs. This theoretical explanation is very important, especially for the readers of the paper to understand about interpreting the graphs. Since graphs consist of topological and typological features, participants are always involved with those things in interpreting graphs. So, this theoretical background is closely related to the research question and methods of the study.

The researchers also present information about previous researches involving graphs (p. 431). This explanation informs the readers about what has been done in the past relating to graphs. The other theoretical backgrounds in the paper are about ethnomathematics and semiotics. It is important to explain ethnomathematics and semiotics because the researchers used them as the theoretical foundation of their study. Ethnomathematics has altered researchers to study the mathematics people actually do and semiotics is concerned with understanding the relation of signs and their referents. The relation between semiotics and the question and method of the study is presented implicitly in the paper (see page 432, first paragraph): “…interpretant signs are what our research participants (verbally, pictorially) produce when they are asked to explain what a graph means or refers to”. So, both of the theory are related to the research question and method.


In my own perspective based on the paper, the researchers use the judgmental sampling as their method in selecting the participants of the study (see p. 432). In this sense, the reason is that the researchers rely on their judgment to select the most appropriate participants of the study based on the research purpose. This sample method is appropriate for an ethnography study and also in line with the research method. This judgmental sampling method is quite natural, requiring the researchers to ask very simple, direct questions about what the participants do or understand about graphs.

The explanation about the background of the participants, including the diversity in gender, previous studies, workplace, years of working experience, and publications, is quite specific. The sample of the study is very small (not a representative sample), only sixteen participants, and does not permit generalization to a large population. In the ethnography study, most researchers have no intention of generalizing the results of their studies. But, the good point is that the researchers made the explanation of participants and situation quite clear which allows for replication of their work in other settings or situations by other researchers. Concerning the ecological generalizability of the research, the researchers clearly state the nature of the environmental situations, the setting, under which a study took place such as the background of the participants, the place of the interview and the order of the graphs presented to the participants. The sample is also appropriate for the purpose of the study because all of the participants are scientists and have at least 5 years experience of conducting research and have received either national or international awards for their publications (p. 432 – p. 433).

Data collection

The researchers used several instruments for colleting the data including videotapes, transcripts, and artifacts (p. 437). The researchers collected the data by interviewing the participants about their understanding of graphs. They asked them to tell as much as they could about familiar and unfamiliar graphs, drawing on all information given including the captions. Some of the good points of the interview based on the paper are the facts that the researchers transcribed the gestures of the scientists and made a video during the interview session (p. 437). By doing so, the quality of the data is increased which also increases the internal validity of the data collection. The researchers also encouraged the participants to say whatever they thought and had in mind when trying to understand and interpret the graphs (p. 437). This activity of recording participants’ thoughts during the interview session, which is also referred to as research reflexivity, can enhance the internal validity of the research. Unfortunately, since the researchers only used an interview as the method to collect the data, they cannot do the data triangulation or methodological triangulation in their study. The researchers also forgot to employ member checking in their study. They did not ask one or more participants in the study to review the accuracy of the transcript interviews.

Using video during the interview is also important with regard to the external validity of the research. By recording all the interview sessions, the other researchers can easily repeat the study using different groups of subjects in different situations. Accordingly, videos also contribute to the ecological generalizability of the research because by recording the interviews, the researchers make clear the setting under the study takes place. Using video is also more reliable than only audio or memorizing. In this sense, the method is independent from the researchers.

The researchers used three different graphs (distribution, population, isoclines graph) in their study (p. 433). The good side of the selection of graphs is that the researchers present the reason of selecting those three graphs: “… the three graphs are common to the literature in ecology and in textbooks on the topic” (see p. 433, paragraph three). The graphs also differ in type and complexity. Unfortunately, there is no any explanation in the paper about the reason why the researcher presented the graphs in this order to the participants during the interview.

Data analysis

In the data analysis, the researchers have a lot of experience relating to the methodology for analyzing graphs (see p. 437, paragraph three): “over the past 9 years, we developed a successful methodology for analyzing graphs, graphing, and graph use from an anthropological perspective…”. The researchers also said that they were reading all the transcripts and viewed all videotape interviews independently (p. 437). It means that they did interrater reliability in analyzing the data. They also negotiated in classifying all cases until they arrived at the same agreement. In this sense, it enhances the internal reliability of the research because both of the researchers discuss the codes of transcripts and the significant events in the videos until they reach the same conclusion. The researchers also explain explicitly that they have worked collaboratively on the research of graphing for many years and argue that this long-term collaboration leads to the strong agreement and high interrater reliability (p. 437).

The facts that the researchers read all the transcripts and viewed all videotapes of the interviews independently are also indications that they did an investigator triangulation. Comparing the findings from two different investigators is good for the consistency of the study. In fact, they coded the transcripts in the same way which is also a good indication of the high interrater reliability and consistency of the study. The researchers also explained the danger of “gone native” in their study. In order to deal with this problem, they employed the techniques proposed by anthropologists that disrupt their common sense (p. 438). This technique is also enhancing the internal validity of the study.

The paper shows the line of reasoning of the researchers in data analysis quite clearly. It presents the way the researchers coded the interview transcripts, analyzed the videotapes of interviews, and even shows the danger of “gone native” as they probably do during the analysis. The readers can follow the argumentation of the researchers in each step of the data analysis quite easily. So, the researchers enhance the external reliability of the study by presenting the tractability or transparency of the research quite clearly.


The authors present the conclusion of the study separately from the discussion. Although it is separately, the conclusion arises from the results and discussion. In the results, the authors present the level of performance of participants with some statistical analysis. They used correct interpretations as criterion variables and have as a result (see page 442, paragraph two): “… scientists who are based at the university or college level tended to be more successful than their nonteaching colleagues, t(14) = 3.88, p = .002”. In this manner, the use of statistics, specifically parametric statistics, is something strange for the ethnography study. This ethnography study uses the judgmental sampling rather than a truly randomized selection which do not allow for inferential statistics. Furthermore, the use of parametric statistics such as t test requires large samples. The authors also elaborate the difference and incorrect interpretations in several sections with some quotes to underpin the results.

In the discussion, the authors present several results from a previous research which is in line with the study. However, the authors also provide their argument which is contradicted with one of the previous researches. In the conclusion, the authors present the answer to their initial question (see page 470 last paragraph): “… when scientists are very familiar with some situation and its graphical representation, the graph (because of its transparency) constitutes an encoding (together with the familiar practices, circumstantial knowledge, etc) worth 10,000 words”.

Consistency: how well do theory, approach, methods, analysis and results fit together?

In my perspective the theory, approach, methods, analysis and results of this study fit together quite well. The authors present the theory (e.g. topological and typological features of graphs, ethnomathematical literature, and semiotics) which underpinned the research quite clearly. For example, the authors stated clearly about the importance of the theory in their study (see page 431, last paragraph): “in this study, we follow the directions and propose a semiotic framework that takes graphs as a multimodal texts that are configured from a number of signs including topological (graphical, pictorial) and typological (mathematical, linguistic) elements”.

Although the authors do not mention explicitly the relation between the theory, research approach, methods, analysis and results, it can be argued that the ethnography approach fits well with the theory underpinning the study. Then, it follows that the interview is a good method for ethnography research. Overall, the flow of the research from the theory, approach, method, analysis to results is in line and presented quite clearly. All of these elements fit together in seeking the same goal, better understanding of how scientists interpret familiar and unfamiliar graphs.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s