Estimated reading time: 11 minutes
Scaling AI agents in financial services has become a decision that starts in the architecture and holds up through governance and operations. Once an agent starts querying sensitive data, calling APIs, summarizing events, or supporting decisions in service, fraud, credit, and compliance, it becomes part of a critical layer of the operation.
The conversation shifts tone: what matters is predictability in production, with an audit trail, access control, visible cost, and reliable integration with the environment the company already has to keep running.
Gartner projects that 40% of agentic initiatives will be canceled by 2027 over control failures and undefined ROI. That’s the same gap Stanford’s AI Index Report 2026 measures at 88% AI adoption against under 10% scaled agent use: everyone adopted, few sustain it in production.
The risks that grow with scale
That scale doesn’t just expand processing capacity. It expands, in the same proportion, specific risks:
- Regulatory exposure: when an agent’s decision has no audit trail;
- Model risk: when it responds with stale data, incomplete lineage, or permissions beyond what’s needed;
- Third-party risk: when an integrator, gateway, or external model becomes an opaque dependency;
- Cost risk: when the bill grows faster than the governance over it.
A case outside financial services gives the measure of the problem at another scale. In a presentation at the Databricks Data and AI Summit, Vikas Ranjan, from T-Mobile, described an operation that processes about 700 TB of raw data daily, with peaks of 2 million records per second.
In financial services, agents don’t operate on neutral data. They touch journeys with material consequences: service, onboarding, fraud prevention, document analysis, support for internal teams, operational observability, and access to regulatory knowledge. Each of these fronts depends on reliable context and clear limits on what the agent can query, infer, and act on.
Agent architecture has to be born with governance
Not every agent task requires open-ended reasoning. In banks, three execution patterns cover most cases, and each answers to a different risk profile.
- Deterministic workflow: no model decision along the way, applied to text sanitization, structured extraction and output formatting. Low risk, high repetition: predictability is what sustains the pattern.
- Sequential orchestration: one step depends on the previous one, with a human review point before the critical action. One agent gathers context, another structures the answer, a third applies compliance policy before releasing it. This is the right pattern when the cost of a mistake is high and the final step can’t be left to the machine alone.
- Concurrent execution: independent tasks running at the same time on the same case, for example document validation, internal policy checks, operational context consolidation and service priority classification. Cuts total time without giving up structure.
All three need to operate inside the corporate infrastructure, inheriting identity, audit and observability from the environment, instead of rebuilding those layers for the agent.
Anthropic documents the same criterion for agent architecture: add complexity only when it demonstrably improves the outcome, because full autonomy trades predictability for latency, cost, and compounding error risk. In banking, that trade has to be an explicit architecture decision.
Best practices for putting agents into production with less friction
One of the strongest recommendations from the T-Mobile case is to design for scale from day 1. That principle gets even more relevant in financial services, where structural changes after growth increase cost, complexity, and risk exposure.
Putting agents into production, then, requires decisions that go beyond picking a model. It takes defining where using an LLM actually adds value, how agents access data, which controls apply during execution, and how the whole system will be observed and operated.
A pragmatic path includes:
- Define where to use an LLM: separate deterministic flows from those that genuinely require interpretation or generation by the model.
- Choose the orchestration strategy: use sequential execution when one step depends on the previous one, and concurrent execution when independent tasks can happen in parallel.
- Connect agents to governed data: centralize catalog, lineage, and access control to define what information each agent can retrieve.
- Build observability from the start: instrument logs, alerts, monitoring, and runbooks before going into production.
- Treat cost as an architecture requirement: account for model consumption, retrieval, storage, and processing during the design of the solution, not just after usage grows.
- Distribute operational knowledge: set aside sprint time for documentation, a knowledge base, and ownership transfer.
That last point matters especially for scale. Teams that operate agents need to reduce dependence on isolated specialists and let new engineers quickly understand the architecture, the critical flows, and the operating procedures.
RAG as a knowledge layer
When an agent needs to work with corporate knowledge, Retrieval-Augmented Generation (RAG) is one of the strategies for feeding updated information to the large language model (LLM) at query time.
Instead of baking all the knowledge into the model’s parameters, the architecture retrieves relevant information from a knowledge base and adds that context to the prompt used to generate the response.
The flow splits into three stages:
- Ingestion and indexing: documents, knowledge bases, and operational data are split into blocks, or chunks. Each chunk is converted into a vector representation by embedding models and stored in a search index.
- Retrieval and reranking: the query combines semantic vector search with lexical search, such as BM25. Reranking models, such as cross-encoders, prioritize the most relevant chunks before sending them to the LLM.
- Grounded generation: the LLM receives the retrieved content as context to generate the response and cites the sources or chunks used as reference.
RAG or fine-tuning for corporate knowledge
We evaluated fine-tuning as an alternative for incorporating corporate knowledge in banking environments, but ruled it out in most of the scenarios we analyzed.
| Criterion | RAG | Fine-tuning |
| Knowledge updates | Updates directly in the knowledge base, no retraining | Requires a new training cycle for every change |
| Answer traceability | Cites the source used at retrieval | Doesn’t reference the source document |
| Permission control | Applied at retrieval time | Doesn’t exist at that level; baked into model weights only |
| Operational cost | Added retrieval latency | Recurring retraining cost |
That choice also comes with trade-offs. The retrieval step adds latency and expands the architecture that has to be operated. Indexes, update pipelines, permissions, retrieval strategies, evaluation, and monitoring all become part of the product.
Where the architecture tends to lose quality
Chunking without considering context
Splitting documents by a fixed count of characters or tokens alone separates information that should stay together. When that happens, retrieval loses context and response quality drops.
Strategies like semantic chunking or parent-child structures preserve relationships between chunks and improve the quality of the material sent to the model.
Access control only at the application layer
Protecting only the interface leaves a gap in the architecture. Authorization rules also need to hold at the retrieval layer.
Role-Based Access Control (RBAC) and Access Control Lists (ACLs) applied during search restrict the retrieved documents according to the permissions of whoever is querying.
In a bank, for example, someone with access to operational reports shouldn’t be able to retrieve chunks from documents restricted to the board.
Content without authorization must not enter the context sent to the LLM.
Stale vector index
Response quality also depends on how current the queried knowledge base is. When a document changes and the index still holds an older version, the agent works with stale information.
Reindexing the entire base on every change drives up cost and processing time. An incremental sync strategy with the Lakehouse updates only the changed data and cuts the need to reprocess the whole set.
Running on the Databricks stack
On the Databricks stack, these responsibilities are split across different components:
- Mosaic AI Vector Search: search and retrieval over the vector index
- Unity Catalog: governance, permissions, and data lineage
- MLflow: evaluation of response fidelity and relevance
- Lakehouse Monitoring: tracking drift and quality over time
Agents and RAG work as production systems, not as an LLM call wired to a prompt.
Final quality depends on the whole chain: data, permissions, retrieval, orchestration, generation, evaluation, observability, and operations.
Designing these layers from the start cuts rework as volume grows and turns an LLM experiment into an architecture that can operate at scale.
Maturity before autonomy
According to the Unlocking Enterprise AI study, by Economist Impact with Databricks (2024), 85% of companies worldwide already use generative AI, projected to reach 99% by 2027. In the same study, 22% consider their infrastructure ready for AI and 37% say their models are ready for production. Adoption ran ahead of the capacity to sustain it.
TreeID developed a maturity self-assessment for adopting AI agents in production. The questionnaire identifies governance gaps, technical dependencies, and operational risk points across four layers: data, integrations, applications, and observability. With the result, the organization defines what to prioritize before expanding agent autonomy.
Frequently asked questions
Production readiness depends on architecture, governance, access control, observability, monitored cost, and integration with existing systems. Without that foundation, the agent is still an experiment.
When the task is repetitive, structured, and low-risk, such as sanitization, simple classification, or output formatting. In those cases, predictability is worth more than flexibility.
It depends on the dependency between steps. Sequential flows work best when there’s human review or policy enforcement before the next action. Concurrent flows work best when independent tasks can run in parallel to cut total time.
The risk lies in who accesses the data, but also in who issued the instruction, which source was used, which policy was active, and which action was executed. Without centralized control, audit and traceability turn fragile.
Cost needs to enter the design from the start. That includes model choice, execution frequency, concurrency, API calls, context retention, reprocessing, and infrastructure policies.
Map the current maturity of data, integration, governance, and operations. Without that diagnosis, the company speeds up the agent’s visible layer while leaving weaknesses in the foundation.
Conclusion
Scaling AI agents starts before choosing the agent. Autonomy only holds up when data, integrations, applications, and observability provide governance, traceability, and clear mechanisms for intervention.
The greater the autonomy, the greater the need to control what the agent accesses, track how it operates, and step in when necessary. In production, scale depends less on more capable models and more on an architecture built to run with security, cost predictability, and control.
Sources
- Stanford HAI. “The 2026 AI Index Report”. Stanford University, 2026.
- Gartner. “Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027”. June 2025.
- Anthropic. “Building Effective AI Agents”. December 2024.
- Economist Impact and Databricks. “Unlocking Enterprise AI”. November 2024.
- Vikas Ranjan (T-Mobile). “Managing Databricks at Scale”. Databricks Data and AI Summit, 2025.