Strengthening Forensic Science
Strengthening Forensic Science A Way Station on the Journey to Justice
“In any legitimate justice system, … truth must play a paramount and integral role…. The very survival of the rule of law depends not only on a justice system that administers the law fairly, but a system that is just by being well-grounded in … truth….[M]ore research is needed in the techniques and science already in use. With the importance of forensic science to truth and justice, the science employed and relied upon by judges and juries must be valid. It does not matter how well forensic scientists abide by testing protocols, or how reliable the techniques are, if the underlying science does not actually reveal what the expert says it does. Method validation studies and new research must be on-going even in the area of traditional forensic disciplines.”1 [Emphases added.]—Kenneth E. Melson, President, AAFS, 2003–2004
Melson’s words, published 6 years before Strengthening Forensic Sciences in the United States: A Path Forward2 was issued saying much the same thing, made me immensely proud to be a member of the American Academy of Forensic Sciences. So much so that I placed a special order for reprints of his essay to show others what the Academy stood for. The absence of any anti-defense-bar attitude in this writing by a career federal prosecutor further strengthened its explicit message: the duty of all in the criminal justice system is to seek out the truth and act on it. Although, because of the inherent asymmetry of the system, this duty falls more heavily on prosecuting attorneys than on defense attorneys, it must be the touchstone for all forensic science practitioners. Their allegiance, our allegiance, must always be to the truth. Although this may seem too obvious to merit stating, this is clearly not the case as long as we have crime lab directors who speak of defense attorneys as “the enemy.” It is not the case as long as anyone questioning an established forensic practice is labeled a “defense hack” or told “well, I don’t know how you stand on law enforcement.” As one whose work in criminal matters is with the prosecution as often as with the defense, I find these attitudes reprehensible and the antithesis of the view set out in the words cited at the top of the page. The counterpart of these attitudes on the civil side is found in forensic practitioners who, at the behest of their employing attorneys, destroy their notes or never create them, in order to avoid writing down a truth that can in some scenario assist the other side.
Melson’s essay appeared as we were beginning to grasp the magnitude of the wrongful convictions brought to light through forensic science and the efforts of the Innocence Project. It was quickly found, again largely through efforts of the Innocence Project, that the wrongful convictions arose from incompetent defense attorneys, unethical prosecutors, misguided reliance on “eyewitnesses,” and flawed forensic testimony, occurring either singly or in combination. Scores, and then hundreds, of persons were discovered to have been convicted of heinous crimes of which they were innocent. Even as this depressing news was being assimilated, we realized that we were seeing but a small fraction of the total: those wrongful convictions discoverable through forensic DNA analysis. Criminal acts leaving no DNA, by far the largest category, are prosecuted using the same procedures and evidence types that led to the known wrongful convictions. For example, eyewitness testimony, in spite of mountains of peer-reviewed studies showing it to be perhaps the least reliable type of evidence, continues to be considered the strongest evidence by most laypersons and many judges. Consequently, it remains an effective means of gaining convictions even in the absence of other evidence and even when the testimony is from a single witness. Unfortunately, it is going to be a long time before this flaw in our justice system can be eliminated, and longer still before consideration will be given to reversing single-eyewitness convictions. Another problem that will be difficult to solve in the short term is the conjunction of incompetent defense work and incompetent forensic testimony. Although appeals claiming insufficient assistance of counsel do get considered, the odds against success for such efforts are very high. This leaves forensic science as the single part of the system amenable to near-term improvement.
[ —Thomas L. Bohan, President, 2009–2010, American Academy of Forensic Sciences. ]
Although the trials known to have led to wrongful outcomes rarely turned on forensic testimony alone, there appears to be a reasonable likelihood that, had the forensic work been correct, the outcomes in those trials would have been different. Therefore, flawed forensic testimony cannot be absolved from blame just because it was not the sole cause of the bad outcome. For the most part, that flawed testimony was delivered by an incompetent or overreaching practitioner. However, the National Academy of Sciences report that arose in part from concern about wrongful convictions quite rightly went further than simply calling for better supervision and certification of forensic practitioners. Stating that many common crime-lab forensic practices had never been scientifically validated, the report called for research to determine which practices were valid and over how broad a range of application the validity existed. It has been noted that during the months immediately following its release, the report did not have a significant effect on criminal trials, including those in which the prosecution was relying on the same types of evidence criticized by the report. This lack of immediate response may be due to the conclusory manner in which the criticisms were framed. Although, given the breadth of the study, this brevity with respect to specific practices is understandable, it means that more review work needs to be done. An illustration of what remains to be done is provided by other National Academy of Sciences reports on forensic practices, such as that regarding the use of polygraphs3 and that regarding the use of trace-metal analysis in bullets.4 Those earlier reports tick off all of the studies claimed to have validated the practice in question, then describe strengths and weaknesses of those studies as support for opining that the practice in question has not been validated. With respect to the pattern-based techniques the latest report criticized, the tabulation of prior studies needs to be done. As it stands, the report’s conclusions about lack of validation have not been accepted by practitioners of the questioned practices, most of whom continue to cite studies that they claim constitute validation. This contrasts, for example, with the effect of the cited National Academy of Sciences report on trace-metal analysis of bullet lead. Once that report issued, there was an immediate and complete cessation of attempts to proffer evidence based on the use of the criticized technique.
The studies that Melson, the National Academy of Sciences, and the American Academy of Forensics Sciences call for will have several benefits in addition to the direct one of establishing the ranges of validity of commonly used forensic theories and techniques. For example, they will help call the attention of judges and trial attorneys to basic forensic facts. What if the Montana judge or defense attorney in the hair-morphology travesty-of-justice trial had possessed the then-widespread knowledge that basic hair morphology is worthless for individualizing hair specimens to a defendant? What if the judge in the Willingham case in Texas5 had been aware that scientific studies had eliminated from fire investigators’ armamentarium the rule-of-thumb analysis of the prosecution’s expert witnesses?
In summary, what is needed immediately is a series of validation investigations. A validation investigation is a threshold study to determine whether a technique or theory the scientific validation of which has been questioned has in fact already been scientifically validated. This is a necessary first step in each instance, given that the practitioners of the respective techniques claim with some heat that their practices have already been validated. In order to secure their needed cooperation in studies aimed at finding the limits of reliability of these practices, a body with the recognition and respect of the National Academy of Sciences must first investigate whether the practice has already been validated, and, if so, what the limits of its validity are. More importantly, once the investigation is complete, this body must publish a definitive report on its conclusions. These threshold studies would not involve any experimentation or lab research, but rather an examination of all the prior studies (especially those reported in the refereed literature), that the practitioners believe have established the validity of their practices.
There will be three possible outcomes to the threshold studies. One, an unlikely one, will be that the practice in question has already been validated. The second, more likely one, is that it has not been validated, at least with regard to determining its range of validity. The third outcome would be, like that of the “bullet-tracing” method study, a finding that the theory or practice was invalid. A particularly important potential example of the latter is the so-called shaken baby syndrome theory. If the critics of this forensic theory are correct, there will be thousands of convictions and plea bargains to be re-examined. Yes, plea bargains, since most such bargains are entered into by defendants convinced that if they go to trial they will be convicted and sentenced to much longer terms than they can obtain through a deal with the prosecutor.
It should be obvious to all that the forensic practices that have come under serious challenge should be subjected to these validation studies. Calling for them is not a defense scheme for getting criminals released or acquitted on a technicality. It is an effort to prevent persons from being wrongfully convicted, to release those who have been, and to redirect law enforcement resources to the apprehension of the actual criminals.
The National Academy of Sciences February 2009 report drew the complaint from some that it had attacked the field of forensic science and, what in a perverse manner carried much the same message, the assertion that it had not attacked forensic science. Given the definition accepted by the AAFS among many others that forensic science is the application of science to questions arising in law, it is difficult to see how it is possible to attack “forensic science.” The report was in fact a criticism of the U.S. justice system for its misuse of science and of the individuals and laboratories who would misuse science. It was an attack on those who use nonvalidated applications of science in prosecuting defendants. It was an attack on those practitioners who claim zero error rate for their work and those who assert that only they are qualified to determine whether their methods are valid.
Another concern raised by those resisting a systemic examination of questioned forensic practices is that this examination will lead to a disruptive witch hunt or fishing expedition (depending on one’s taste in metaphors). This is far from being the case. Although scientists in different forensic specialties may be able to add a few practices to the list, I believe that it is fairly complete in terms of those practices that have been identified by the National Academy of Sciences and others as needing evaluation, that is, those that have come under questioning that is not frivolous. Some examples of the practices needing threshold studies follow:
- Bite mark analysis used to identify perpetrators who have bitten their victims, a method the limits of which have never been delineated and which probably in the hands of some practitioners is applied far beyond the extent for which any validation exists.
- Tool mark analysis, an extremely broad field, parts of which can be subject to validation but nevertheless appear never to have been studied for this purpose.
- Handwriting comparison analysis to determine whether two specimens were written by the same person or whether a specific person had written a particular specimen is a field that since the handing down of the Daubert
decision emphasizing the need for quantification of reliability has been carrying out a series of tests to establish the different aspects of reliability in the described undertakings. In spite of this continuing research, there are some practitioners in the field who claim levels of reliability that have not yet been established.
- Friction ridge analysis for purposes of identification and in particular latent fingerprint analysis where a small fractional print from a crime scene is compared with databases of full fingerprints, seeking a match that will place a particular individual at the scene, is a traditional forensic discipline holding one of the longest pedigrees. Unfortunately, that long history has been used by many of its practitioners to resist calls for quantification. Contrary to the straw men erected by some of these practitioners, calls for quantitative studies do not hinge on a suggestion that fingerprints are not unique. The studies needed are those of the probability, however one wishes to define it, that, as the practitioners progress from the discovery of a fractional latent print to the identification of that print with a particular individual, an error is made, a false positive occurs. Although initially the Federal Bureau of Investigation was the ideal entity to carry out such studies, providing it published its results in the open refereed literature, I believe that this is no longer the case. Its resistance to the very idea that such studies are needed will cast a shadow on whatever it now undertakes. Unlike those techniques that must be evaluated to determine whether they have already been validated, latent fingerprint identification needs basic research to determine its limitations with respect to latent-print quality and quantity, and some measure of the probability of false positives—the type of work that academic research teams are suited for, providing they can get access to the large fingerprint databases now in existence. Nevertheless, an evaluation by the National Academy of Sciences or an equivalent entity of the work that has already been done would provide a valuable boost to any program for additional research.
- Shaken baby syndrome theory, by whatever name it morphs into, which holds that a small dead child displaying certain limited soft tissues pathologies but no other injuries was shaken to death by the last person who held the child when he or she was conscious. Of all the questioned forensic theories and practices, this is the one I would put at the top of the list for the threshold examination. Of all the currently questioned practices, this is the only one asserted by a respectable minority of specialists to be completely invalid. While a vitriolic dispute continues between the pro and con groups, defendants continue to be sentenced to exceedingly long prison terms based on the theory. While the theory’s supporters accuse the doubters of profiting from defending baby-killers, the doubters label the supporters as zealots lacking any knowledge of physics or of the scientific method. It is long past time that an authoritative body outside the adversarial system examined the underpinnings of the theory and published its results.
As people of good will across the criminal justice system have grappled with how to implement the strengthening of forensic science called for by the National Academy of Sciences, I have been struck by the composition of those invited to partake in this crucial work. In particular, I have been struck by the near-absence of scientists from this group. I have heard lobbyists without the slightest idea of what scientific research consists of call for “research but not so that it interferes with our solving the most important problems.” I have had the experience of being in a roomful of people discussing the form legislation addressing scientific issues in the forensic arena should take and realizing that I was the only person present who had spent any time at all doing scientific research. It seems obvious that a broad swath of scientists should be engaged in examining each forensic technique about which serious questions have been raised. In determining the degree, for example, that handwriting comparison analysis has already been validated there should of course be an expert in that field. However there must also be statisticians and indeed scientists and engineers familiar with the examination of evidence. As stated above, this study and the threshold studies in other areas will not involve direct research of the type needed to validate the practice in question. Rather, they will be examinations of the professional literature in the field, including especially that put forth by practitioners to support the proposition that validation has already taken place. Until an authoritative body such as this has reached a conclusion as to what has already been validated and what has not, very little progress is likely in the basic validation research. Those tests that will ultimately be indicated are nontrivial undertakings and will not be undertaken until those who can carry them out are convinced of their need.
This is a crucial period with respect to forensic science in this country. The 2009 National Academy of Sciences report on forensic science in the United States has opened an opportunity for beneficial change that will soon be gone, gone for the foreseeable future. We must take advantage of it to erect the framework now that will instill and ensure a continuing robustness throughout all of forensic science. The result of this work will be to the immeasurable benefit of us all, because it will be of benefit to the American system of justice.
- 1 President’s Editorial—The Journey to Justice, J Forensic Sci, July 2003, Vol. 48, 705.
- 2 National Research Council, The National Academies, National Academies Press, Washington, DC, 2009.
- 3 The Polygraph and Lie Detection, National Research Council, The National Academies, National Academies Press, Washington, DC, 2003.
- 4 Weighing Bullet Lead Evidence, National Research Council, The National Academies, National Academies Press, Washington, DC, 2004.
- 5 For one account of this tragic case, see TRIAL BY FIRE, by David Grann, page 42, New Yorker, September 7, 2009.