The movement to systematize, even automate, the assessment of bodies of evidence has some beneficial features that have attracted influential proponents:  ensuring comprehensive inclusion of relevant studies; seeking to evaluate studies solely on their methodologic quality rather than subjective impressions; and providing detailed descriptions of the process used to draw conclusions so that it can be replicated by others. But there has been substantial collateral damage to informed interpretation of evidence and application to policy, particularly when algorithms such as the GRADE and ROBINS approaches are imposed.

• Arbitrary hierarchies of data quality are assigned without consideration of the applicability to a particular topic, with some studies predetermined to be inherently superior to others (e.g., randomized trials over observational studies, cohort studies over case-control studies);
• Individual studies are scored in isolation and the contributions are considered in isolation rather than integrated across lines of evidence to address compensating strengths and limitations, referred to as “triangulation”;
• The benefit of deep subject matter expertise is set aside in favour of simplistic algorithms in an attempt to be “objective,” which often makes the assessment uninformed regarding the nuances that are needed for interpretation;
• The application of algorithmic evidence evaluations to policy fails to recognize that policy decisions reflect an informed judgment, calling for a fully informed consideration of the research that is impeded by the constraint of an algorithm.

The list of ill-informed and likely harmful decisions reached in large part through the application of GRADE and other similar approaches continues to grow. When an arbitrary algorithm leads to judgments that differ from the collective wisdom of the full range of subject matter experts, it is possible but unlikely that the consensus is wrong, but more likely that the algorithm has failed to capture insights that the experts are able to provide.  Science, policy, and public health require thoughtful evaluation of evidence including informed debate among to make progress and determine the best use of the information available at a given point in time. Nothing less makes the grade.

The DE-grading Epidemiology (DEEP) Network has been formed to encourage open debate about the use of algorithms to assess epidemiological studies, and maintain contacts between those of us who are working on these issues. It will pursue strategies to improve our ability to methodically and accurately integrate and interpret epidemiologic evidence.