Error of the Day & Maintaining Integrity of Algorithmic Results

If you’re into algorithms, you should absolutely subscribe to the MIT Technology Review newsletter called The Algorithm.

Earlier this month, the folks at The Algorithm asked “what is AI, exactly?” The answer is reproduced below.

The question may seem basic, but the answer is kind of complicated. In the broadest sense, AI refers to machines that can learn, reason, and act for themselves. They can make their own decisions when faced with new situations, in the same way that humans and animals can.

As it currently stands, the vast majority of the AI advancements and applications you hear about refer to a category of algorithms known as machine learning. These algorithms use statistics to find patterns in massive amounts of data. They then use those patterns to make predictions on things like what shows you might like on Netflix, what you’re saying when you speak to Alexa, or whether you have cancer based on your MRI.

Machine learning, and its subset deep learning (basically machine learning on steroids), is incredibly powerful. It is the basis of many major breakthroughs, including facial recognitionhyper-realistic photo and voice synthesis, and AlphaGo, the program that beat the best human player in the complex game of Go. But it is also just a tiny fraction of what AI could be.

The grand idea is to develop something resembling human intelligence, which is often referred to as “artificial general intelligence,” or “AGI.” Some experts believe that machine learning and deep learning will eventually get us to AGI with enough data, but most would agree there are big missing pieces and it’s still a long way off. AI may have mastered Go, but in other ways it is still much dumber than a toddler.

In that sense, AI is also aspirational, and its definition is constantly evolving. What would have been considered AI in the past may not be considered AI today. 

Because of this, the boundaries of AI can get really confusing, and the term often gets mangled to include any kind of algorithm or computer program. We can thank Silicon Valley for constantly inflating the capabilities of AI for its own convenience.

It’s good to be reminded of this definition as we contend with the latest releases of the legal research databases as the databases continuously tweak their underlying algorithms — the latest being Westlaw Edge.

With Westlaw Edge comes a revised “WestSearch Plus.”

Introducing the next generation of legal search. Get superior predictive research suggestions as you start typing your legal query in the global search bar.

WestSearch Plus applies state-of-the-art AI technologies to help you quickly address legal questions for thousands of legal topics without needing to drill into a results list.

We’re starting to see a time when the Google Generation is already predisposed to not drill into a results list and now the databases are actively advocating for the users to blindly rely on the top result in the list.

Along with the consequences of fake news on algorithmic results when using Google, for example, we must also be aware of the errors within the legal research databases themselves. To that end, a fellow law librarian, Mary Matuszak, has been collecting the errors that she finds during the legal research process in the various databases and distributes them via the Law-Lib listerv as “Error of the Day.”

From October 30, 2018:

Error of the Day  A  Lexis typo (possibly scanning error) in  Excessiveness of Bail in State Cases, 7 A.L.R.6th 487.   The following group of letters is used six times throughout the document, CocainesepBail.   A quick look at the Westlaw version shows that it should be Cocaine – Bail

From November 5, 2018:

In the Case People v Kindell, 148 AD3d 456 (1st Dept 2017), Susan Axelrod is listed as both the counsel for the Appellant and the Respondent.   The official version, the print, does not list the attorneys.


I confirmed with ADA Axelrod that she did not represent the defendant and opposing counsel was not someone with the same name.   I also checked the defendant’s brief and it lists Ms. Moser as counsel.

While these errors are seemingly minute individually, the consequences are greater in the aggregate.

My own mentor, a law librarian who had been in the profession for 40 years, kept a print file of the errors that he found in the databases while performing legal research. The file was overflowing by the time I saw it roughly 3 years before his retirement.

Because an algorithm’s results are only as good as the underlying data, as we move toward an algorithmic society that relies heavily on algorithmic decision making, these errors could have consequences on the development of the law.

Reserve the Student Wellness Room Today!

The Law Library in conjunction with the Dean of Student’s Office and the Student Wellness Advisory Committee is very proud to announce the availability of the Wellness Room. This space is designed for students to decompress, de-stress, and rejuvenate.

Wellness Room

The Wellness Room includes:

  • Comfortable Seating
  • Aromatherapy
  • Yoga Mats
  • Mini Zen Garden
  • Sound Machine
  • Coloring Books
  • Wellness Collection

Wellness Room Guidelines:

  • The Wellness Room booking defaults to 30-minutes blocks of time.
  • The Room may be reserved for 1-hour maximum per person, per day.
  • No more than 3 students allowed at one time.
  • The Room should be straightened after use.
  • The Room is intended for TTU Law Student use.
  • Associate Dean Chapman may enter the Wellness Room at any time.
  • If you are finished with the Wellness Room before your reservation has expired, please stop by the Reference & Information Desk and inform the staff so they can adjust the reservation.

To reserve the Wellness Room, go to the Law Library’s Room Reservation link for additional information.

Practitioners Rank Legal Research as Only Top-20 Specific Legal Skill for the “Whole Attorney”

In a recent survey conducted by the Institute for the Advancement of the American Legal System (IAALS), a wide array of legal employers ranked the legal skills and professional competencies and characteristics that they believe new lawyers most need to succeed. (There is a detailed accounting of the study’s results and an explanation of the study’s role within IAALS’s broader project in the summer 2018 edition of The Bar Examiner, pp. 17-26.) The results revealed that legal employers value foundational characteristics and competencies much more than they do foundational legal skills. 

The 20 Foundations Identified as Most Necessary in the Short Term for New Lawyers 

• Keep information confidential

• Arrive on time for meetings, appointments, and hearings

• Honor commitments

• Integrity and trustworthiness

• Treat others with courtesy and respect

• Listen attentively and respectfully

• Promptly respond to inquiries and requests

• Diligence

• Have a strong work ethic and put forth best effort

• Attention to detail

• Conscientiousness

• Common sense

• Intelligence

• Effectively research the law 

• Take individual responsibility for actions and results

• Regulate emotions and demonstrate self-control

• Speak in a manner that meets legal and professional standards

• Strong moral compass

• Write in a manner that meets legal and professional standards

• Exhibit tact and diplomacy

As noted by The Bar Examiner, [t]he only specific legal skill that reached the top 20 was legal research.

What we’re seeing is a serious dissonance between what legal educators (and by extension law students) and legal practitioners think are the most important skills for practice. Most law students are graduating from law school thinking that they have the skills necessary to practice as attorneys, but that opinion is not shared by the profession they hope to enter. In one survey, 95 percent of hiring partners and associates believed that recently graduated law students lacked key practical skills at the time of hiring. In another survey, 76 percent of third-year law students believed that they were prepared to practice law “right now,” while only 23 percent of practicing attorneys believed that recent law school graduates were ready to do their jobs.

Interestingly, the 77 foundations identified [in the IAALS Survey] as necessary for new graduates are largely the same across all workplaces, which means that as we begin to identify the overarching learning outcomes that we can—and should—expect of a legal education, we have at least one common goal: the whole lawyer.

When breaking out just the specific legal skills necessary for practice, the IAALS Survey revealed the following:

• Effectively research the law

• Understand and apply legal privilege concepts

• Draft pleadings, motions, and briefs

• Identify relevant facts, legal issues, and informational gaps or discrepancies

• Document and organize a case or matter

• Set clear professional boundaries

• Gather facts through interviews, searches, document/file review, and other methods

• Request and produce written discovery

• Effectively use techniques of legal reasoning and argument (case analysis and statutory interpretation)

• Recognize and resolve ethical dilemmas in a practical setting

• Conclude relationships appropriately

• Critically evaluate arguments

• Maintain core knowledge of the substantive and procedural law in the relevant focus area(s)

• Prepare client responses

• Draft contracts and agreements

• Interview clients and witnesses

The article goes on to mention the possibility of legal educators and professionals using the information from this survey to regulate the skills new lawyers truly need when they enter practice.

One thing that becomes clearer with every practitioner survey is the importance of legal research skills for practice.

Use Google Scholar’s Advanced Search for Narrow Case Law Searching

There are various tricks to using Google Scholar for free case law searching that will help you narrow your search results to relevant cases.

Carole Levitt — nationally recognized author and speaker on internet legal research — notes that to narrow your search results, make sure to use the Advanced search menu.

To use the Advanced search menu for case law searching, click on the case law radio button on the Scholar homepage. Then, click the menu icon to find the Advanced search menu.

Screen Shot 2018-05-13 at 9.25.53 PM
While the first four Boolean connector and phrase search boxes located on Scholar’s Advanced search menu are the same as’s Advanced Search menu, there are three “field” search boxes and one drop-down list unique to Scholar’s Advanced Search menu.

Screen Shot 2018-05-13 at 9.27.49 PM
Unfortunately, many searchers don’t use these features because Scholar never bothered labeling them with appropriate case law terminology—they simply left the articles’ database labels on them. For example, the field search box labeled as:

  • Return articles authored by should really be labeled Return cases authored by
  • Return articles published in should really be labeled Return cases published in
  • Return articles dated between should really be labeled Return cases dated between

If you didn’t know that the “Return articles authored by” field search could actually be used to “Return cases authored by,” you might simply enter the judge’s name (e.g., Charles Vogel, a California judge), into the “all the words” Boolean search box and limited your search to California. With this search, it’s possible your results could also be that of a party, an attorney, a witness, or an expert, and so on, many of which would be irrelevant to your query.

But restricting the judge’s name to the “Return articles authored by” field search box eliminates all those extraneous and irrelevant results that you would have had to sift through.

The Art of Selecting Cases to Cite

A question that comes up pretty often from law students in legal writing classes is, “how do I know which cases to cite?”

This is a good question and one that really depends on the cases that are found during the research process. An article by Douglas K. Norman titled “The Art of Selecting Cases to Cite” in 63 Tex. B. J. 340 (Apr. 2000) does a wonderful job of discussing how to select cases to cite.

From the article:
“Most legal writers seem to have developed an instinct for which cases to pick and which to discard. Moreover, the considerations that the writer consciously or subconsciously brings to bear on which cases to include in legal citation are more complicated than merely citing to the most recent case from the highest court. These considerations probably begin with an informal categorization of all the cases found in the process of researching a given issue.”

The article goes on to discuss categorizing cases. “The cases supporting a given proposition of law might roughly be categorized as follows:

Seminal Case–This is generally the first case from the highest court to have decided the issue and stated the proposition of law in question. If the proposition was itself a reversal or revision of earlier authority, the seminal case is the reversing or revising case. As the first case to have stated the proposition in question, the seminal case has generally gone into some depth in analyzing the issue and the court’s rationale in a manner that might not be repeated in later cases.

Parroting Cases–With a common proposition of law, numerous cases will have simply parroted the language of the seminal case, adding little or nothing to the analysis.

Bolstering Cases–In addition to parroting seminal authority, the bolstering case adds new reasoning and analysis to support the underlying proposition of law.

Reformulation Cases–These cases take the proposition of the seminal case and either restate it in different terms or refine the analysis in some way that may be more or less helpful to the reader. When, for example, the seminal case was written in the legalistic jargon of the past, the reformulation case may delete the jargon and restate the proposition in plain English.

Pseudo-Seminal Case–When the seminal case has been forgotten or lost in the chain of citation, a more recent case will often emerge as the one most courts presently cite as the oldest or most reliable case to support the given proposition. This case effectively takes the place of the lost seminal case.

Companion Cases–As I use the term here, companion cases (not to be confused with the more technical use of this term) are parroting cases that have over time been so consistently cited together with the seminal (or pseudo-seminal) authority that they achieve a certain perceived legitimacy and it would now seem awkward to break the habit of citing the companion case together with the seminal authority.

Parallel Cases–Occasionally, separate lines of authority for the same proposition develop without any common source; or, perhaps more likely, the original source or seminal case is buried so far back in the chain of citation that it has been all but forgotten. This then leads to two or more lines of cases standing for the same proposition, with different courts typically preferring one or the other of the parallel lines of authority.

Storehouse Cases–It sometimes happens that, when there are multiple parallel cases with no clear seminal or pseudo-seminal case to which they all refer, somewhere down the road a particular case will attempt to collect or “storehouse” all of the parallel lines. If this storehouse case is reliable, it is a prime candidate to be cited from then on as pseudo-seminal authority.

Application Cases–Some cases that have only marginal value as support for an abstract proposition of law, have great value in their application of the proposition to facts similar or analogous to the facts of your own case.”

After categorizing cases, it is time to select the cases to cite. The article offers guidelines for selection:

  • Provide Both Seminal and Recent Authority
  • Generally Omit Parrots
  • Use Vertical Strings To Show Continuity Within a Court
  • Use Horizontal Strings To Show Continuity among Courts
  • Reconcile Parallel Lines of Authority
  • Use Bolstering and Reformulation Cases to Strengthen and Better Explain
  • Use Application Cases For Similarity and Analogy

Selecting the proper cases is part of the underlying analysis required of the legal research process. As the author notes, improper case selection may cause the reader to reject the underlying legal proposition, which could be the death knell for a case.