Notice: This article was created with AI.
What’s It About?
Under the name “Dayjob: Tax”, Surge AI has published a benchmark that examines how well AI agents solve tasks from professional tax practice. According to Surge AI it consists of 50 realistic assignments, among them individual tax returns and international tax cases. For these, according to Surge AI, 1,789 documents are available, such as PDFs, Excel worksheets and handwritten receipts.
The result is sobering: according to Surge AI, the best model tested reached a pass rate of 10.8%. Most of the work produced would accordingly not have stood up to a professional review.
Background & Context
According to the account by Surge AI, the benchmark goes beyond answering fixed, predefined questions. The agents were supposed to identify problems on their own and make well-founded decisions without being given every step. That includes noticing contradictions in the documents and picking up points that clients did not raise at all.
The bar is deliberately set high. Surge AI works with a 95% rubric for a pass. That is meant to reflect whether a result would be reliable enough to be used in communication with clients. As examples of errors, Surge AI names an agent that in several attempts overlooked a deadline on which a claim to tax relief could depend. Another agent repeatedly made false assumptions about the family status of clients. The source material was only available in shortened form, so further details could not be taken into account.
What Does This Mean?
- According to the information from Surge AI, merely working through documents is not enough for tax advisory. What is decisive is judgment and the recognition of unspoken problems.
- The strict 95% rubric from Surge AI shows that very high reliability would be needed for use in client communication. Errors with deadlines or personal circumstances can have noticeable consequences.
- For law and accounting firms and for companies, the result argues for regarding AI agents as support with human review for the time being. No statement about other types of task or later model generations can be derived from it.
Sources
Dayjob: Tax – evaluating AI agents on 50 professional tax assignments (Surge AI)
Top AI scores 10.8% on tax work, exposing professional judgment gap (aigovernance.com)
Dayjob: Tax – evaluating AI agents on 50 professional tax assignments (Beitrag auf daily.dev) (daily.dev)
Dayjob: Tax – Benchmark-Seite (Surge AI)
Nachrichtenbeitrag zur Dayjob-Tax-Benchmark (OpenAI Hub)
LinkedIn-Beitrag: Dayjob Tax – Benchmarking AI agents on professional activity (LinkedIn)
Surge Dayjob Tax Benchmark – Kurzüberblick (AI-TLDR)
Surge AI Dayjob Tax Benchmark: 50 Tax Assignments – Blogbeitrag (explainx.ai)
Hacker-News-Diskussion zu Dayjob: Tax (Beitrag 50275626) (Hacker News)
Hacker-News-Diskussion zu Dayjob: Tax (Beitrag 50282067) (Hacker News)
This article was created with AI and is based on the listed sources as well as the language model’s training data.
Further Reading: The Future of AI