In a rigorous study aimed at determining whether AI has progressed to where it can navigate the sort of critical thinking required for legal education Stanford Law Professor Julian Nyarko, along with co-authors, Yale Law Professor Sarath Sanga and Stanford Liftlab’s Researcher, Alenjandro Salinas, have reached some potentially unsettling conclusions. Using a clever research design, the study’s authors asked 40 participants to draft questions, and their own answers, that might be asked by students in a law school Contracts class. These questions were then given blind to other law professors and to various AI LLM’s. each of whom (which?) wrote answers. The question drafters were the asked to evaluate the quality of these answers and 75% of the time the AI answers were preferred. Moreover, in only 3.5% of cases were AI answers judged potentially harmful to students compared to 12% of answers provided by actual professors. The authors suggest that these LLM’s are proving able to synthesize complex material and perform well even when there are strong arguments on two sides of a question. Whether students will actually learn as well from AI answers is, of course, another question. Stay tuned.