Panos Ipeirotis (NYU Stern School of Business) has published this illuminating account of building a stack of AI agents to assess his students’ oral presentations. He has a detailed, step-by-step account of his motivations, what he did to build the system, and an after-action assessment of what worked well and what worked poorly. His takeaways include this:

Take-home exams are dead. Reverting to pen-and-paper exams in the classroom feels like a regression. In our case, we wanted to check that the students who worked in the team projects actually contributed and understood what they submitted; we would not be able to do that with pen-and-paper exams in the classroom.

We need assessments that evolve towards formats that reward understanding, decision-making, and real-time reasoning. Oral exams used to be standard until they could not scale. Now, AI is making them scalable again.

A comment, below the jump, on what law professors might take away here.

Combining Professor Ipeirotis’s work with AI agents and that of Professor Chandler with Linux, which I posted about earlier today, I am motivated to wonder:

What is the basic skill set that a field (discipline, school, department) should expect of a faculty member, now and into the future?

When PowerPoint became a thing for college and university faculty, more or less 25 years ago, I taught myself PowerPoint. When the World Wide Web became a thing, I taught myself elementary HTML and, later, PHP. I am no Excel expert, but I manage well enough. Along the way, I asked for and received low-level IT support from inside my school and my university, but it never occurred to me to wonder whether I was taking on a role or responsibility beyond what should have been expected from a (then junior) faculty member. I was solving problems in my own classroom and my own research experience, problems arising at the intersection of then “conventional” practice and “novel” tools and techniques.

Does the appearance of Generative AI prompt a reconsideration of answers to contemporary versions of those problems? Should it be up to me, in other words, to figure out how to use and deploy whatever AI systems I prefer to use or to avoid? As before in my experience, I might call on IT help desk support and/or input from our university’s central center for teaching support.

If the answer to my question might be “no,” what then? The university or the law school (or any law school, or academic center or department) might “up its game,” providing more and better training and support. Here at Pitt, that is largely what is happening. I observe and hear about related patterns emerging at many other schools and universities.

But there are other options. Should the teaching profession re-consider the long-standing assumption that faculty autonomy in the classroom defaults also to faculty responsibility for pedagogy-plus-subject-matter-expertise-plus-sociotechnical design? I have co-taught courses with other full-time faculty. I have never co-taught a course with a sociotechnical expert. I don’t know what that might look like.

I recall that when Harvard Law School launched its H2O project, I was told that part of the vision was having a classic law professor lead the in-person classroom in tandem with an in-person technical colleague at a podium in the room, who would “steer” the audio-visual experience for students, anchored in the H20 materials, in tandem with the classroom conversation.

Exploring this option (in my own mind, at least) leads to my wondering whether schools (and by extension, professional services organizations of many sorts) might imagine investing in a tier of technical experts who are compensated fairly for providing a kind of “faculty collaborator” status, without faculty accountability for student outcomes (or lawyer accountability for client results). In law firms, this would involve a pivot from “paralegal” or “legal assistant” roles; in law schools, a pivot from “faculty support” or “law librarian” roles.

Maybe this is happening already, and the news hasn’t reached Western Pennsylvania? That’s been a problem in the past.