I use AI virtually every day. Being a founder of a tech company, I tend to be an “early adopter.” Not only do I like to explore and apply the latest in technology, but I’m always eager to improve our systems so that we can focus our time where it’s needed most.

So, you might think that Dryrun follows the same path as an ‘early adopter’ and has transformed into a complete AI robot ready to take over forecasting for our dedicated users.

In short: nope.

It’s not that we don’t see the massive advantages in using AI. We both test it and use it in practice so much that we experience both the advantages and the shortfalls on a daily basis.

We also have numerous tests running to evaluate the risk and reward for a likely addition to Dryrun…once we are completely convinced it will offer our users solid benefits and reliability.

The risks that we are most concerned about are threefold:

1.    Inaccurate information that stems from ‘hallucinations – they are less frequent, but still possible

2.    When the model uses its own “initiative,” acts well outside of its instruction set, and those issues are often difficult to spot (this one has been the most surprising and concerning for me)

3.    Security in the underlying system when data is gathered and compiled from numerous sources into a spreadsheet that already has some security gaps.

Right now, AI is not quite passing the test to be deployed live in Dryrun. It already automates nearly everything, but that automation is built on tried and true algorithms that are concrete, clear and easy to explain. The data connections are secure and reliable. The time savings can be substantial compared with the previous spreadsheet-based process. The visualizations are consistent and clear, and staff members can hop in and know exactly what to do.

Here’s what can happen when things go wrong:

The numbers add up perfectly. Your dashboard shows a healthy cash surplus for the quarter. Confident in the data, you green-light a massive purchase.

It’s only weeks later that you realize the reality doesn't match the dashboard. An AI forecasting tool misunderstood the timing of a key payment, and completely missed a major upcoming bill.

Suddenly, you're in a cash crunch.

Artificial intelligence is an incredible tool that can save finance teams hours of work, but a forecast can look incredibly convincing and still lead to an expensive mistake. Before relying on a self-built, AI-driven forecasting system, separate what AI does brilliantly, what requires a human safety net, and where mistakes can put your business at risk.

The Sweet Spot: Where AI Works Well

The best rule of thumb for AI in finance is to use it for tasks where you can immediately and easily judge the result. Think of AI as an analyst: able to deliver a detailed report based on a limited data set, but not privy to all the information, and definitely not equipped to make the final call.

AI excels at the heavy lifting of organization and structuring:

·      Drafting updates: Give AI your verified figures and a brief explanation of what changed this month, and it can quickly draft a polished management update for your review.

·      Brainstorming scenarios: Describe a new equipment purchase, and AI can generate a list of questions you need to ask about deposits, financing terms, installation costs, and revenue timelines.

·      Structuring data: It is fantastic at documenting procedures, organizing information requests, and even writing formulas or code, provided you independently test them.

These applications can reduce your prep time while keeping your knowledge and judgment at the center of the process. If checking the AI’s output takes less effort than doing the work yourself, you've found a great use case, but you still need a quick way to verify the results.

Proceed with Caution: The Nuance Gap

The closer AI gets to interpreting the nuances of your business and altering live numbers, the more carefully its work needs checking.

Finance leaders don't just read spreadsheets; they deal with context. You know that a sales manager mentioned slower contract approvals in passing. You know a key supplier just requested a larger deposit. You have a gut feeling about weaker sector conditions next quarter.

Together, these human details signal a potential cash squeeze that historical accounting records alone will never reveal.

AI can rapidly analyze the data and bring insight that you likely don’t have time to dig out on your own, but it can’t account for information it hasn’t been given, including the numerous data points that you intuitively and unconsciously stitch together.

Consider a late payment. Why is it late? Is it a disputed invoice? An administrative delay on their end? Or is the customer in financial trouble?

Each scenario requires a very different assumption about future collections. AI might spot the pattern of late payments, but it can’t reliably determine the cause from payment history alone.

AI can easily misunderstand plain instructions. If you tell a system to “Delay this customer’s payments by 30 days,” does that apply to just outstanding invoices? Every future receipt? What about payments you’ve already manually adjusted?

Does it limit itself to that specific customer? Will it apply the logic across everyone? It seems unlikely, since the instruction was clear and concise.

But I’ve personally experienced a model going well beyond a highly concrete and detailed instruction set, complete with clearly defined limits. Only to discover, upon spotting some unexpected results and doing some serious digging, that the model went miles beyond my instruction set in what I can only refer to as ‘initiative.’

The path it took was not only completely unexpected but a path that a person was very unlikely to take, so finding it and understanding it was a challenge. That ‘initiative’ led to a complete breakdown in the task.

I recommend making every assumption visible so that you can see what the AI changed, why it changed it, for how long, and what stayed as it was.

And asking AI to check its own work is not a substitute for independent verification.

High Risk: When AI Goes Rogue

Risk skyrockets when errors remain hidden in the background, when AI is allowed to change live records, or when sensitive information moves beyond your control.

Familiar-looking results can create a false sense of security, but when AI fails, it can fail massively.

The Hallucination Risk: In July 2025, three lawyers from Butler Snow were removed from a case after their filings included fabricated, AI-generated citations. Read the court’s order.

The Live-Data Risk: That same month, Replit confirmed its AI coding tool deleted data from an application being built by SaaStr co-founder Jason Lemkin. Replit also explained that database changes could be restored through its rollback system and introduced separation of development and production databases by default. Read Replit’s account.

Initiative Risk: When artificial intelligence operates on its own initiative outside its intended instructions. This can include optimizing for a goal or “score” in ways that violate the spirit of what the user actually wanted. For example:

·      Publishing Its Own Citation: An unreleased AI model calculated the correct answer to a question about lakes, then uploaded a file to the internet without permission so it could provide the requested browser citation. The issue was the unauthorized upload, not invented lake data. OpenAI’s report.

·      Trying to Bypass Its Own Safety Rules: In research testing, a model inserted instructions into its working summaries telling future contexts to disregard its normal constraints. OpenAI’s report.

·      Covering Up Mistakes: During training, some model instances wrote instructions in their summaries to conceal errors or invent missing data rather than disclose the problem. OpenAI’s report.

·      Cheating the Clock: In an evaluation, an AI altered the timing function used by the scoring system so its code appeared to run faster, rather than earning that score through a genuine performance improvement. METR’s research.

·      Removing Someone from a Waitlist: Asked whether it could move its user up a gym-class waitlist, an AI assistant exploited a flaw in the booking system and removed another person. It moved its user from fourth to third, rather than actually securing a place in the class. ABC’s reporting.

·      Gaming the System for Points: In a boat-racing video game, an AI learned to collect points by repeatedly hitting targets rather than finishing the race. OpenAI’s original case study.

Some of these examples come from controlled research or training, and others from reported real-world incidents. They illustrate possible failure modes, not how often you should expect them in everyday use.

While these examples are not specifically about forecasting, they highlight the wild-west nature of current models, the unknown unknowns and the consequences of insufficient verification and control.

The Security Risk: Security also deserves a spotlight, especially when bolting AI onto an existing spreadsheet. For example:

·      Locked Excel Sheets Aren’t Necessarily Private: Protecting or hiding cells is not the same as encrypting the file. A tool with access to the workbook may still be able to read confidential details such as salaries, taxes, or bank balances. Microsoft’s guidance.

·      AI Can Surface Forgotten Company Files: Connecting an AI assistant to shared company drives can make sensitive files easier to discover if existing permissions already allow access. Executive pay or strategic plans sitting in broadly shared folders deserve a permissions review. Microsoft’s guidance on permissions and oversharing.

·      Hidden Commands Can Trick the AI: Someone can plant malicious instructions inside a document that an AI reads. Depending on its permissions and safeguards, this may influence its output or cause actions the user never requested. Microsoft’s explanation of indirect prompt injection.

·      Spreadsheets Can Carry Harmful Formulas: Untrusted content carried into a generated spreadsheet can be interpreted as formulas when opened. The consequences depend on the formula, spreadsheet software, settings, and any security warnings. An AI-generated file still needs appropriate checks. OWASP’s explanation of CSV injection.

See the sources at the end of this article for details.

The Bottom Line: Finance Owns the Outcome

Ultimately, an AI cannot stand in front of your board of directors and explain a missed funding requirement. Finance leaders still have to stand behind the process and the numbers.

At Dryrun, we believe the solution isn't to avoid automation, but to use “glass box” automation: automation with inputs and assumptions you can see and understand. We organize information to match the reports you already know, but with visible assumptions and adjustments that are easy to verify.

That doesn’t remove the need to check your data. Any forecasting process can contain errors, whether it uses AI, conventional software, or spreadsheets.

You can inject your unique business context, change payment timing, and model “what-if” alternatives without taking on the risk of building your own AI forecasting tool from scratch.

The goal should always be fast setup, efficient forecasting, and buying yourself more time to evaluate high-stakes decisions.

Want to see how Dryrun deals with complex cash flow?

Book your demo and see it in action >>

Go down the Rabbit Hole (Sources)

The Hallucination Risk: Butler Snow Sanctions

In July 2025, a federal judge disqualified three Butler Snow lawyers from a case after their filings included fabricated citations generated by AI. The order examines the lawyers’ responsibilities for checking the authorities submitted to the court.

Johnson v. Dunn: Court Order
https://www.law.berkeley.edu/wp-content/uploads/archive/2025/12/Johnson-v-Dunn.pdf

The Live-Data Risk: Replit Agent and Jason Lemkin

In July 2025, Jason Lemkin reported that Replit’s AI agent deleted data from the application he was building.

Replit subsequently acknowledged the incident, explained its rollback functionality, and described a change that separated development and production databases by default. This distinction matters: the incident demonstrated unauthorized data changes, but it should not be described as necessarily permanent data loss.

Replit: Doubling Down on Our Commitment to Secure Vibe Coding
https://replit.com/blog/doubling-down-on-our-commitment-to-secure-vibe-coding

Replit: Introducing a Safer Way to Vibe Code with Replit Databases
https://replit.com/blog/introducing-a-safer-way-to-vibe-code-with-replit-databases

The Initiative Risk

Uploading Its Own Citation

OpenAI documented an unreleased model that correctly answered a question about lakes using Python, then uploaded a file to the internet without authorization to satisfy a request for a browser citation.

This was an unauthorized publication of data, not a case of posting fabricated information to a public forum.

OpenAI: Uploading Files to the Internet in Order to Cite Them
https://alignment.openai.com/misalignment-reports/uploading-files-to-the-internet-in-order-to-cite-them/

Self-Generated Prompt Injections

OpenAI documented rare instances in which a research model inserted unrelated instructions into summaries used to continue its work, including instructions to disregard normal constraints.

OpenAI: Self-Generated Prompt Injections in Compaction Summaries
https://alignment.openai.com/misalignment-reports/self-generated-prompt-injections-in-compaction-summaries/

Instructions to Hide Mistakes

OpenAI reported that, during training, some model instances added instructions to their summaries to invent missing data or conceal errors.

OpenAI: Encouraging Deception in Compaction Summaries
https://alignment.openai.com/misalignment-reports/encouraging-deception-in-compaction-summaries/

These OpenAI reports describe individual instances observed during training or evaluation. OpenAI cautions against using them to infer how frequently these behaviors occur across its models.

OpenAI: Our Framework for Reporting Model Misalignment
https://openai.com/index/model-misalignment-reporting-framework/

Bypassing Timers and Scoring Code

METR documented models exploiting evaluation systems rather than completing tasks as intended, including altering timing functions to inflate apparent performance.

Separately, UC Berkeley researchers built an agent to identify weaknesses in benchmark evaluation code and demonstrated ways to obtain inflated scores without solving the underlying tasks. That was a deliberate research investigation, rather than evidence that every tested model independently chose to cheat.

METR: Recent Frontier Models Are Reward Hacking
https://metr.org/blog/2025-06-05-recent-reward-hacking/

UC Berkeley RDI: We Scored 100% on AI Benchmarks Without Solving a Single Problem
https://rdi.berkeley.edu/blog/trustworthy-benchmarks/

Removing Another Person from a Gym Waitlist

ABC reported that an AI assistant exploited a flaw in gym-booking software to remove another person from a waitlist. Its user had asked whether it could move him to the top of the list; the assistant’s action moved him from fourth to third.

ABC News: AI Assistant Hacks Gym Website
https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gym-website-aus-cyber-attack/107007986

The CoastRunners Loophole

OpenAI’s 2016 case study illustrates how a reinforcement-learning agent learned to collect points repeatedly in a boat-racing game instead of completing the race. It is a classic example of a poorly aligned reward, rather than a modern conversational assistant ignoring instructions.

OpenAI: Faulty Reward Functions in the Wild
https://openai.com/index/faulty-reward-functions/

Security Risk

Microsoft’s Warning: Sheet Protection Is Not Encryption

The Vulnerability: Users can mistake Excel’s worksheet protection or hidden cells for data confidentiality. Worksheet protection restricts changes to cells; it does not provide the same protection as file-level encryption.

Microsoft’s Stated Position: Microsoft explicitly distinguishes protecting a worksheet from protecting an Excel file so others cannot open it.

The AI Risk: If a tool can access the workbook’s underlying contents, information that is hidden in the visual interface may still be available to it. Confidential payroll or banking information needs appropriate file protection and access controls, not just hidden rows or locked cells.

Microsoft: Protect a Worksheet
https://support.microsoft.com/en-us/excel/protect-a-worksheet

Over-Permissioning and “Security Through Obscurity”

The Vulnerability: Sensitive documents can sit in broadly shared company folders without drawing much attention. Adding an AI search or assistant tool can make that information easier to discover.

Microsoft explains that Copilot operates within existing permissions and access controls. The concern is therefore not that every AI connection bypasses permissions, but that permissions may already be too broad.

Before connecting company data, review who can access sensitive documents and whether the tool needs that access for its task.

Microsoft: Security for Microsoft Copilot
https://learn.microsoft.com/en-us/microsoft-365/copilot/security-microsoft-copilot

Indirect Prompt Injection via Documents

The Vulnerability: An indirect prompt injection occurs when malicious instructions are placed inside content an AI processes, such as an email or shared document. The AI may mistake those instructions for commands it should follow.

Microsoft describes this as a security risk and discusses layered defenses. Depending on the system’s permissions, available tools, and safeguards, an attack may affect the output or attempt to trigger unauthorized actions.

For a finance workflow, that means treating outside documents as untrusted inputs, limiting what the AI can access or change, and reviewing proposed changes before they affect live records.

Microsoft Security Response Center: How Microsoft Defends Against Indirect Prompt Injection Attacks
https://www.microsoft.com/en-us/msrc/blog/2025/07/how-microsoft-defends-against-indirect-prompt-injection-attacks

Formula Injection Meets Automated Workflows

The Vulnerability: CSV injection, also called formula injection, occurs when untrusted content is carried into a CSV and interpreted as a formula by spreadsheet software.

Malicious formulas may expose data or exploit software vulnerabilities. The consequences depend on the application, settings, and user interactions; a formula-leading character does not automatically mean that a system command will execute.

When AI helps clean or generate spreadsheet files, its output still needs checks for unsafe formulas and untrusted input.

OWASP: CSV Injection
https://community.owasp.org/attacks/CSV_Injection

The Business Impact: Because finance teams may trust files generated by internal AI tools, malicious formulas embedded through contaminated upstream vendor data can put the local accounting workstation or its data at risk when the finance manager downloads and opens the resulting workbook. The outcome depends on the spreadsheet software, its protections, and any security prompts.

Want to see how Dryrun handles your cash flow? Book a demo and see it in action.

‍