Background and Importance

Most research articles undergo a peer review process that can involve adding information, clarifying key points, checking for completeness, acknowledging limitations, or ensuring the article is presented in the preferred style of the journal. Although peer review has been in place for centuries, its efficacy is unclear. A 2002 meta-analysis failed to demonstrate any overarching benefit from peer review, but studies that examine specific components of the peer review process provide support for its utility. Wagner and Middleton demonstrated that technical editing, defined as any process designed to “improve accuracy or clarity or impose a predefined style, “increase[d] the readability and quality of articles”; however, the investigation failed to specify the exact editorial processes involved. A 2002 study in this journal concluded that the creation of methodology reviewers resulted in more focused editorial comments, and a second study showed that these comments resulted in modest improvements in article quality.

Despite graphs and figures being commonplace in the biomedical literature, studies demonstrate that published graphs are often suboptimal and fail to use the potential of the format. The peer review process has been shown to do little to correct these shortcomings.

Annals of Emergency Medicine has a comparatively sophisticated and well-studied peer review process relative to other journals. Papers are initially assessed by one of approximately 40 decision editors, who send approximately 50% of submissions for further review. Since 1997, each article sent for peer review has also been reviewed by a dedicated methodology or statistical editor. In addition, beginning in 2008, a senior “graphics” editor was assigned to review each paper. In 2008, this review took place at final acceptance and was the last step in the peer review process before publication. In 2011, the process had changed so that the graphics editor saw the paper before the first request for revision, and his comments were included along with those of the other peer reviewers. During both periods, the graphics editor had access to all comments made by other reviewers.

Goals of This Investigation

In this paper, we examine how the characteristics of graphs changed from submission to publication and from year to year, with a particular focus on whether the introduction of a graphics editor resulted in increased critique of graphs and improved graphic data presentation.

Study Design

We conducted an observational time-series analysis of the quality of data graphs in Annals from 2006 to 2012. We created a sample of 180 randomly selected published research articles composed of 60 manuscripts initially submitted for publication in 2006, 60 submitted in 2009, and 60 submitted in 2012. This represents approximately 60% of original science articles published in these years. These 3 years represent 3 distinct periods in the development of the journal’s editorial process: the period before formalized graph review (2006), graph review after the paper was accepted for publication (2009), and graph review at first “revise and resubmit” request (2012).

We separated each figure, including its caption, from the paper and selected a maximum of 5 graphs per paper. If a paper included more than 5 graphs, we randomly selected 5. In this report, we use “figure” to designate an object that is listed as a figure in the paper and “graph” to indicate a depiction of study data in a nontext, nontabular format. As such, figures such as flow diagrams of patient enrollment, theoretical models, or photographs are not graphs. Regardless of whether graphs were labeled as a single figure (eg, Figure 5 A , B , and C ) or were treated separately (eg, Figures 5, 6, and 7), we scored them as a single graph if they shared at least 1 identical axis and were thematically connected and as separate graphs if they were not.

Data Collection and Processing

We conducted the 3 parts of our data collection process independently: measuring the size of each graph, scoring the quality of each graph, and cataloguing the peer review comments about each graph. For the first part, an assistant, blinded to the purpose of the study and the year of each graph, calculated the area occupied by graph and caption by measuring the height and width in centimeters, using the measurement tool in Adobe Acrobat X Pro 10.1.16 (Adobe Systems, San Jose, CA).

For quality scoring, we trained 5 raters on a standardized set of graphs, and, after achieving adequate interrater reliability on a 20-graph sample (percentage agreement >90% on all 85 items), they scored the content and quality of study graphs presented in isolation, separate from their paper. Each rater scored a balanced number of submitted and published graphs from each of the years. Raters, who were never asked to score both the submitted and published versions of the same graph, were blinded to graph year, but not to whether each graph was from an unpublished or a published article because of differences in formatting that could not be blinded.

Raters scored each figure with a standardized form that captured both the graph’s type (eg, bar graph, scatter plot, parallel line plot) and 84 attributes, which included items used to calculate the numerator of the data density index (how many pieces of information are presented per square centimeter of graph), the dimensionality, and items about special features (eg, the depiction of clustering), graph completeness (eg, the presence of appropriate titles and labels), visual clarity (eg, the absence of excessive overlapping), and a gestalt assessment of whether the graph was self-explanatory ( Appendix E1 , available online at http://www.annemergmed.com ). After completing the data form, raters were allowed to refer to the whole paper to confirm that they had correctly understood any abbreviations and had interpreted the purpose of the graph correctly. The data collection form (Excel 2010; Microsoft Corp, Redmond, WA) contained explanations on how to score each item and pull-down menus showing scoring options ( Appendix E1 , available online at http://www.annemergmed.com ). Raters were encouraged to discuss with the authors any scoring decisions for which they were uncertain. During the comparison of published and unpublished versions of a graph, one author (C.D.) had the opportunity to review the scoring of both graphs side by side and correct (in consultation with D.L.S. or R.C. when necessary) any errors.

We created the portmanteau variables “completeness” and “clarity” to indicate whether all elements of these constructs were present and “special features” to indicate whether any such feature was present ( Appendices E1 and E2 , available online at http://www.annemergmed.com ). To be deemed “complete,” a graph had to include 10 essential features, listed and described in full in the scoring form ( Appendix E1 , available online at http://www.annemergmed.com ). A graph with visual “clarity” did not display any of 13 common “visual problems,” which are also listed and defined in the scoring form.

We calculated the data density index, the number of pieces of information per square centimeter, using a modified version of the formula proposed by Tufte. In brief, the data density index denominator is the area of the graph and its caption in the journal. Unpublished graphs were assigned the area of the published ones. The numerator was the number of pieces of discrete, unique information contained in a graph. A properly identified bar in a bar graph received 2 points, 1 for the height of the bar and 1 for the bar’s identifier. Each point in a scatter plot received 3 points, 1 for each axis value and 1 for the linkage of those 2 values. In all graphs, points were awarded for additional unique information such as the quantity depicted by an axis, labels, annotations, regression lines, and statistics (see Schriger et al and Tufte for details). For example, the data density index numerator for Figure 1 , a bar graph, was calculated as 18 bars plus 18 bar identifiers plus 1 for the labeled, distance-dependent x axis, or 37 points in total. The data density index for Figure 1 is 0.42 (37/86 cm ² ).

The complexity of the data graphs, stratified by year and submitted manuscript versus published paper. There are less simple graphs and more intermediate and complex graphs in published papers compared with submitted manuscripts. The number of complex graphs submitted was highest in 2006, which may reflect that papers in later years had categorical outcomes or less complex data structures. This graph has a data density index of 0.42 (37/86 cm ² ).

Determining the numerator of data density index for Figure 2 , a series of univariate plots, is slightly more complicated. The calculation can be understood as follows: the approximately 540 visible points in the 6 rightmost sections (the “All years” plots are redundant and are not counted) are multiplied by 3 (1 for the axis value, 1 for the color, and 1 for the shape); to this we add 8 column identifiers, 32 median lines, 24 medians depicted by words or letters, 6 shape and 3 color identifiers, and an axis title, for a total of 1,694 pieces of information and a data density index of 7.5. Figures 3 and 4 have data density indexes of 1.07 (102/97 cm ² ) and 0.76 (100/132 cm ² ), respectively, and Figure 5 has a data density index of 0.95 (263/277).

The pattern of data density graphed on a log scale for each study year comparing graphs in submitted manuscripts and published papers. Each point represents 1 graph, with the color indicating the complexity of the graph (simple, intermediate, or complex; see text for definitions) and the shape indicating whether the graph was deleted from the manuscript (x), was added during the revision process (+), changed format (eg, changed from pie chart to scatter plot) (Δ), or was unchanged or underwent minor modifications during the review process (•). The location along the y axis of these bolded words in the “All years” panel indicates the geometric mean data density index for each category; the locations of their first letters (O, D, C, A and P) do the same for the by-year data. The thick black line represents the geometric mean data density index for each stratum (submitted or published). The thin colored lines represent subgroup means based on graph complexity. The mean data density index for published graphs was approximately 1 piece of information/cm ² . However, individual graphs had data density indexes ranging from 0.03 to 97 piece of information/cm ² . Several patterns emerge: (1) mean geometric mean data density index increased each year for both submitted manuscripts and published papers, although mean data density index was lower in 2012 because of the absence of very-high-density graphs; (2) data density index and geometric mean data density index increased from submission to publication; (3) simple graphs with low data density indexes tended to be deleted or changed to a different format and added graphs tended to have higher complexity and data density index; (4) graph complexity was highly correlated with data density index. This graph has a data density index of 7.5 piece of information/cm ² .

The “Commented” bar indicates the number of comments made about the data graphics by each type of reviewer for each year. The “Changed” bar indicates the number of comments that led to a change. The numbers within each bar represent the percentage of comments that resulted in change for each reviewer type, with the value at the end of each bar reflecting the overall percent changed for that year. Over time there were more comments, largely because of the addition of the graphics editor, and a larger percentage of comments resulted in changes. The data density index of this graph is 1.07 (102/97 cm ² ).

Bars represent the types of comments made by the different types of reviewers in each year. The number in parentheses indicates the total number of comments depicted by the bar. “Major changes” include deleting or adding a graph or changing its format (eg, bar to box plot). “Conceptual” comments refer to issues about the organization of the graph, including failure to depict clustering, failure to depict the full dimensionality of the data, or failure to show the number of subjects represented by a symbol. “Visual clarity” refers to issues related to the appearance of the graph such as overlapping symbols and improperly labeled or scaled axes. “Minor clarity” refers to issues such as ambiguous or unnecessary abbreviations and failure to follow standard conventions for histograms and box plots ( Appendix E4 , available online at http://www.annemergmed.com ). The graphics editor, who received the paper at a later stage in the review process and knows what the other reviewers have written, tends to make more comments about clarity issues with the graph. The methodology reviewers make the most comments about conceptual issues because their percentages are highest and they make the most comments overall. The data density index of this graph is 0.76 (100/132 cm ² ).

Characteristics of data graphs by study year.

We calculated dimensionality by determining the number of characteristics of each data point depicted in the graph. For example, a simple scatter plot would have a dimensionality of 2 because both the x and y axes provide a unique characteristic. However, if the points were color coded to define a particular attribute, the dimensionality would be 3, and, if a symbol or shape were used to convey another attribute, dimensionality would be 4 ( Appendix E3 , available online at http://www.annemergmed.com ). We considered bar, line, and pie charts to be “simple” graphs; box plots, histograms, and receiver operating characteristic curves to be “intermediate” graphs; and scatter plots, survival curves, parallel line plots, parallel coordinate plots, maps, and hybrid graphs to be “complex” graphs.

We analyzed the effect of editorial comments on the quality of graphic presentation by collecting all editorial feedback to authors during the peer review process from the journal’s database (Editorial Manager; Aries Systems, North Andover, MA). This correspondence included comments from the decision editor, the regular content reviewers, the methodology and statistics reviewer, and, depending on the study phase, the graphics editor. One author (C.D.) used the “find” function of Microsoft Word (version 2011; Microsoft Corp) to highlight graphic and statistical terms, such as “graph,” “figure,” “plot,” “bar,” “scatter,” “box,” “axes,” and “labels.” Using the highlighted terms as a guide, she carefully scanned all text for comments about the manuscript’s graphs. She then attributed each comment to 1 or more of the paper’s graphs, assigned the topic of the comment to one of 16 categories in 4 major classes ( Appendix E4 , available online at http://www.annemergmed.com ), and noted the role of the person who made the comment (regular reviewer, methodology/statistical reviewer, paper editor, or graphics editor). When a sentence in a review contained concepts that could be attributed to several categories, she parsed that sentence into multiple distinct comments. The fidelity of this process was confirmed by independent coding of a sample of reviews by a senior author (D.L.S.).

Primary Data Analysis

We sought to describe differences in a variety of measures of graph quality between original and published versions of articles and among study years. We programmed Stata (version 14; StataCorp, College Station, TX) to compare each element in the original and published versions of each graph in the context of this inventory of figure-related comments. The difference in medians and its CI were calculated with the cid command in Stata (version 14). For graphs that appeared in only the manuscript or published article, we examined whether comments were responsible for the elimination or addition of the graph. Through this process, we determined how often changes in graphic presentation occurred in response to editorial feedback and how often editorial feedback resulted in a change.

Study Design

We conducted an observational time-series analysis of the quality of data graphs in Annals from 2006 to 2012. We created a sample of 180 randomly selected published research articles composed of 60 manuscripts initially submitted for publication in 2006, 60 submitted in 2009, and 60 submitted in 2012. This represents approximately 60% of original science articles published in these years. These 3 years represent 3 distinct periods in the development of the journal’s editorial process: the period before formalized graph review (2006), graph review after the paper was accepted for publication (2009), and graph review at first “revise and resubmit” request (2012).

We separated each figure, including its caption, from the paper and selected a maximum of 5 graphs per paper. If a paper included more than 5 graphs, we randomly selected 5. In this report, we use “figure” to designate an object that is listed as a figure in the paper and “graph” to indicate a depiction of study data in a nontext, nontabular format. As such, figures such as flow diagrams of patient enrollment, theoretical models, or photographs are not graphs. Regardless of whether graphs were labeled as a single figure (eg, Figure 5 A , B , and C ) or were treated separately (eg, Figures 5, 6, and 7), we scored them as a single graph if they shared at least 1 identical axis and were thematically connected and as separate graphs if they were not.

Data Collection and Processing

We conducted the 3 parts of our data collection process independently: measuring the size of each graph, scoring the quality of each graph, and cataloguing the peer review comments about each graph. For the first part, an assistant, blinded to the purpose of the study and the year of each graph, calculated the area occupied by graph and caption by measuring the height and width in centimeters, using the measurement tool in Adobe Acrobat X Pro 10.1.16 (Adobe Systems, San Jose, CA).

For quality scoring, we trained 5 raters on a standardized set of graphs, and, after achieving adequate interrater reliability on a 20-graph sample (percentage agreement >90% on all 85 items), they scored the content and quality of study graphs presented in isolation, separate from their paper. Each rater scored a balanced number of submitted and published graphs from each of the years. Raters, who were never asked to score both the submitted and published versions of the same graph, were blinded to graph year, but not to whether each graph was from an unpublished or a published article because of differences in formatting that could not be blinded.

Raters scored each figure with a standardized form that captured both the graph’s type (eg, bar graph, scatter plot, parallel line plot) and 84 attributes, which included items used to calculate the numerator of the data density index (how many pieces of information are presented per square centimeter of graph), the dimensionality, and items about special features (eg, the depiction of clustering), graph completeness (eg, the presence of appropriate titles and labels), visual clarity (eg, the absence of excessive overlapping), and a gestalt assessment of whether the graph was self-explanatory ( Appendix E1 , available online at http://www.annemergmed.com ). After completing the data form, raters were allowed to refer to the whole paper to confirm that they had correctly understood any abbreviations and had interpreted the purpose of the graph correctly. The data collection form (Excel 2010; Microsoft Corp, Redmond, WA) contained explanations on how to score each item and pull-down menus showing scoring options ( Appendix E1 , available online at http://www.annemergmed.com ). Raters were encouraged to discuss with the authors any scoring decisions for which they were uncertain. During the comparison of published and unpublished versions of a graph, one author (C.D.) had the opportunity to review the scoring of both graphs side by side and correct (in consultation with D.L.S. or R.C. when necessary) any errors.

We created the portmanteau variables “completeness” and “clarity” to indicate whether all elements of these constructs were present and “special features” to indicate whether any such feature was present ( Appendices E1 and E2 , available online at http://www.annemergmed.com ). To be deemed “complete,” a graph had to include 10 essential features, listed and described in full in the scoring form ( Appendix E1 , available online at http://www.annemergmed.com ). A graph with visual “clarity” did not display any of 13 common “visual problems,” which are also listed and defined in the scoring form.

We calculated the data density index, the number of pieces of information per square centimeter, using a modified version of the formula proposed by Tufte. In brief, the data density index denominator is the area of the graph and its caption in the journal. Unpublished graphs were assigned the area of the published ones. The numerator was the number of pieces of discrete, unique information contained in a graph. A properly identified bar in a bar graph received 2 points, 1 for the height of the bar and 1 for the bar’s identifier. Each point in a scatter plot received 3 points, 1 for each axis value and 1 for the linkage of those 2 values. In all graphs, points were awarded for additional unique information such as the quantity depicted by an axis, labels, annotations, regression lines, and statistics (see Schriger et al and Tufte for details). For example, the data density index numerator for Figure 1 , a bar graph, was calculated as 18 bars plus 18 bar identifiers plus 1 for the labeled, distance-dependent x axis, or 37 points in total. The data density index for Figure 1 is 0.42 (37/86 cm ² ).

Only gold members can continue reading. Log In or Register to continue

Share this:

• Click to share on Twitter (Opens in new window)
• Click to share on Facebook (Opens in new window)

Related

Related posts:

Adolescent Male With Right Shoulder Pain After Football Injury Association Between the Opening of Retail Clinics and Low-Acuity Emergency Department Visits Emergency Department Visits Without Hospitalization Are Associated With Functional Decline in Older Persons Yet Another Sedating the Agitated Patient: A Moving Target? Classified 2017 Advertising Rates & Information