Self-Assessing AI Efficiency

When an attorney says "AI has made me more efficient", is that a reliable statement? There's good reason to think it may not be. As professionals whose stock and trade is evaluating the reliability of evidence, we definitely need to consider that possibility.

Two data points (both with their own reliability issues):

🖥️ - "[D]evelopers expected AI to speed them up by 24%, 𝒂𝒏𝒅 𝒆𝒗𝒆𝒏 𝒂𝒇𝒕𝒆𝒓 𝒆𝒙𝒑𝒆𝒓𝒊𝒆𝒏𝒄𝒊𝒏𝒈 𝒕𝒉𝒆 𝒔𝒍𝒐𝒘𝒅𝒐𝒘𝒏, 𝒕𝒉𝒆𝒚 𝒔𝒕𝒊𝒍𝒍 𝒃𝒆𝒍𝒊𝒆𝒗𝒆𝒅 𝑨𝑰 𝒉𝒂𝒅 𝒔𝒑𝒆𝒅 𝒕𝒉𝒆𝒎 𝒖𝒑 𝒃𝒚 20%." This was one of the conclusions of a study, not about lawyers, but about LLM usage among open-source software developers. (Link to the article.) The study concluded that AI use 𝐬𝐥𝐨𝐰𝐞𝐝 developers down by 19%, but the developers 𝐭𝐡𝐨𝐮𝐠𝐡𝐭 it had sped them up by 20%. The study was published in July 2025 and so the data is from even earlier. You could, therefore, make a good argument that developer tools/knowledge have gotten better. But that doesn't change the point I'm focusing on that the participants were 𝗯𝗮𝗱 𝗮𝘁 𝗮𝘀𝘀𝗲𝘀𝘀𝗶𝗻𝗴 𝘁𝗵𝗲𝗶𝗿 𝗼𝘄𝗻 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 when evaluating their own AI usage.

✒️ - "Lawyers are terrible at estimating how long it will take them to do things." This is a quote from me from five minutes ago. 😇 It is, however, based on 20+ years working as a lawyer and with other lawyers. Still, that's admittedly not a scientific study. I will only add that, in addition to my personal experience with actual attorney estimates (including my own), it has also been my experience that this is one of the things that has made lawyers resistant to fixed-fee arrangements.

I'm not arguing that AI isn't making lawyers more efficient (or that it won't in the future). Nor do I think that the debate over that question at its full generality ("Does AI make lawyers more efficient?") is particularly relevant. AI is here in the legal world, and it ain't going anywhere.

To me though, it is an important question when it comes to the particulars. That is, it's pretty important to know 𝐞𝐱𝐚𝐜𝐭𝐥𝐲 𝐰𝐡𝐞𝐫𝐞 AI is and isn't making lawyers more efficient. That informs where law firms should be investing LegalTech dollars, where they should be encouraging (mandating?) AI usage, and conversely where they should be discouraging (prohibiting?) it.

And if I were the one making those decisions, I wouldn't be relying solely on lawyers reporting that they 𝐭𝐡𝐢𝐧𝐤 it makes them more efficient. I'd want data.