AI
AI agents in a holding company: what works today and what does not
What AI agents do reliably in a holding company today, where they fail, and six rules that make them safe on group data, with evidence from 2025 and 2026.
- Published
- 9 min read
- 9 min read
An AI colleague is useful today for narrow, well-defined tasks on verified data. It answers a question about a metric and shows the source, drafts the weekly review, watches for anomalies and reads documents. It is not ready to run long chains of work alone or to take irreversible actions. What separates a useful agent from a risky one is the rules around it, more than the model.
In short
- Agents have improved fast. The Stanford AI Index 2026 reports a rise from 12% to about 66% task success on a benchmark of real computer tasks. They still fail roughly one attempt in three on structured benchmarks.
- Use is wide, value is not. In McKinsey’s 2026 survey nearly nine in ten organisations use AI regularly, and 37% see any effect on earnings.
- Short, well-defined tasks work. Long autonomous chains and irreversible actions do not work yet.
- Six rules make an agent safe on a group’s data: certified data with sources, the user’s own permissions, read-only access by default, outside text treated as data, logs, and tests before every release.
What an AI colleague is
A chat tool answers from what the model remembers. An AI agent can also use tools: run a query, read a document, prepare a draft. OpenAI’s guide to building agents defines agents as "systems that independently accomplish tasks on your behalf".
Anthropic draws a useful line in Building effective agents. Workflows follow "predefined code paths". Agents "dynamically direct their own processes and tool usage". Most recurring work in a holding, such as the weekly review or a reconciliation, is better built as a fixed workflow with AI steps inside it than as a free agent. The same text advises the simplest solution that works, and warns that autonomy brings "higher costs, and the potential for compounding errors".
What the evidence says about reliability
The progress is real. The Stanford AI Index 2026, published in April 2026, reports that agents went from 12% to about 66% task success on OSWorld, a benchmark of real computer tasks. It adds that they "still fail roughly 1 in 3 attempts on structured benchmarks".
Task length is the best predictor of failure. METR measured this on data from March 2025. Models had an almost 100% success rate on tasks that take a person less than four minutes, and succeeded less than 10% of the time on tasks that take more than about four hours. The length of tasks that agents complete with 50% reliability had been doubling about every seven months.
Office work looks similar. TheAgentCompany, a benchmark from Carnegie Mellon University and other institutions that imitates a small software company, found that the best agent completed 30% of tasks on its own (version of September 2025).
Two more results matter for managers:
- An agent that succeeds once may fail the next time. The τ-bench study of 2024 measured whether an agent succeeds on every one of several repeated attempts, and found the models of that time inconsistent. The numbers are dated. The lesson is not: test repeatedly.
- Answers from your own documents can still be wrong. Stanford researchers tested legal research tools that retrieve documents before they answer, an approach known as RAG. In 2024 two of the tools gave incorrect information in more than 17% of queries and a third in more than 34%. Their conclusion: "even RAG systems are not hallucination-free".
Use is ahead of value. In McKinsey’s State of AI 2026 survey of 1,719 respondents, published on 25 August 2026, nearly nine in ten report regular use of AI in at least one business function, about two in ten are scaling agents, and 37% attribute at least some effect on EBIT to AI. Official statistics are more modest: Eurostat counted 20.0% of EU enterprises with ten or more employees using AI in 2025. And Gartner predicts that over 40% of agentic AI projects will be cancelled by the end of 2027 because of rising costs, unclear business value or inadequate risk controls.
What works today in a holding, and what does not
The table is our reading of the evidence above.
| Task | Does it work today? | What the person does |
|---|---|---|
| Answering a question about a certified metric, with the source | Yes, if metrics are defined once and the agent may say it does not know | Asks and checks the source |
| Drafting the weekly business review | Yes, as a draft | Edits and approves |
| Watching metrics and flagging anomalies | Yes | Decides what to do |
| Reading contracts and invoices and pulling out fields | Yes, when the source passage is shown | Checks the exceptions |
| Running a long process end to end, such as the monthly close | Not yet | Keeps the process and gives the agent single steps |
| Payments, corrections to the books, messages that leave the company | It only prepares them | Approves every action |
Six rules that make an agent safe on a group’s data
1. Certified data only, with the source. Someone has to mark which data is official. Microsoft describes this for Power BI: "Certification means that the content meets the organization’s quality standards", and only reviewers chosen by the administrator may certify.
The effect on accuracy is large. An early benchmark by data.world (2023) found that a model answering business questions directly on SQL databases was right 16% of the time, and 54% of the time when the questions were posed over a knowledge graph of the same data. In dbt Labs’ own test of April 2026, a semantic layer reached 98.2% and 100% accuracy for two models, against 90.0% and 84.1% for free-form text-to-SQL. Where the semantic layer could not answer, it said so and returned no number.
Every answer should also point to its source. This can be built today: Anthropic’s Citations feature returns "the exact passages that support each claim".
2. The user’s own permissions, enforced by the data systems. An agent must see only what the person asking may see. OWASP’s guidance on excessive agency is direct: "Implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not." In a group this has a second meaning. Anthropic’s framework for trustworthy agents warns that agents "might inappropriately carry sensitive information from one context to another". One company’s data must not surface in another company’s answers.
3. Read-only by default, and a person approves every action. Anthropic says of its own coding agent: "It has read-only permissions by default". OpenAI’s guide asks for human oversight of actions that are "sensitive, irreversible, or have high stakes", and names cancelling orders, authorising large refunds and making payments.
4. Text from outside is data, not commands. An agent that reads emails, supplier documents or websites can be steered by instructions hidden in them. OWASP calls this indirect prompt injection. It happens when a model "accepts input from external sources, such as websites or files" (OWASP LLM01).
5. Everything is logged. Who asked, what the agent computed, what it answered and what it did. OWASP’s advice on this point: "Log and monitor the activity of LLM extensions and downstream systems".
6. A fixed set of test questions before every release, and permission to say "I don’t know". Because an agent can pass once and fail the next time, it is tested on the same questions with known answers before each update. An agent that may decline to answer is safer than one that always produces a number.
Risk frameworks on one page
Three documents give a common language for AI risk. None of them binds a company in Azerbaijan, Uzbekistan or Kazakhstan, but each is a good checklist.
- The NIST AI Risk Management Framework (United States, released on 26 January 2023, voluntary). Its core has four functions: govern, map, measure and manage. NIST says the framework is being revised.
- ISO/IEC 42001:2023, a management system standard for AI published in December 2023. It fits organisations of any size that develop or use AI.
- The EU AI Act, Regulation (EU) 2024/1689. For high-risk systems it requires effective human oversight, awareness of automation bias, a way to stop the system, and automatic logs (Articles 12 and 14). Most uses in a holding, such as reports, forecasts and drafts, are not high-risk under the Act. Check the dates when you read about it: according to the European Commission, an amending package that entered into force on 27 July 2026 moved the rules for high-risk uses in sensitive areas to 2 December 2027.
Risk is also what holds companies back. In McKinsey’s survey on AI trust, published in March 2026, nearly two-thirds of respondents named security and risk concerns as the top barrier to scaling agents.
The regional context
Azerbaijan, Uzbekistan and Kazakhstan have each adopted a national AI strategy, and data sits near the top of them.
- Azerbaijan. The Artificial Intelligence Strategy for 2025 to 2028, approved by Presidential Order No. 530 of 19 March 2025, makes improving data management and infrastructure a priority and includes the move to data-driven management. There is no AI law yet.
- Uzbekistan. The Strategy for the development of AI technologies until 2030, approved by Presidential Resolution No. PP-358 of 14 October 2024, aims to bring AI-based software products and services to 1.5 billion US dollars. A decree of 22 October 2025 adds a target of at least 100 AI projects by the end of 2026.
- Kazakhstan. The Concept for the development of AI for 2024 to 2029, approved by Government Resolution No. 592 of 24 July 2024, names data as its first direction. The Law on Artificial Intelligence of 17 November 2025 requires AI output to be labelled and directs restricted-access data to local AI systems.
Where personal data is involved, each country’s storage rules apply to agents too. We compare them in the article on personal data storage in Azerbaijan, Uzbekistan and Kazakhstan.
How to start
Start with one narrow task and measure it against a baseline. OpenAI’s guide recommends an incremental approach, and Anthropic advises the simplest solution that works. The McKinsey survey shows where the value comes from: nearly three-quarters of high performers fundamentally redesigned their workflows because of AI, against one-quarter of the others.
A sensible first task has three features. The data under it is already in order, which is the subject of our article on management reporting in a group of companies. A person checks the output before it is used. And success is measured: accuracy on the fixed set of questions, hours saved each week and the cost of one answer.
How Sahman approaches this
Sahman is at an early stage, and our AI colleague Samira works on demo data today. Samira answers from certified metrics with the user’s own permissions, shows the source, may answer "I don’t know", and asks for approval before anything leaves the company. Questions that need rows with personal data go to an open model inside the country. The details are on the Samira page.
Sources
- 1A practical guide to building agents. OpenAI, 2025
- 2Building effective agents. Anthropic, 2024-12-19
- 3The 2026 AI Index Report. Stanford Institute for Human-Centered Artificial Intelligence, 2026
- 4Measuring AI Ability to Complete Long Tasks. METR, 2025-03-19
- 5TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. Frank F. Xu and co-authors, arXiv, 2025-09-10
- 6τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Shunyu Yao and co-authors, arXiv, 2024-06-17
- 7AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries. Stanford HAI, 2024-05-23
- 8The state of AI in 2026: On the road to ROI. McKinsey & Company, 2026-08-25
- 920% of EU enterprises use AI technologies. Eurostat, 2025-12-11
- 10Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027. Gartner, 2025-06-25
- 11Promote and Certify Power BI Content with Endorsement. Microsoft Learn, 2025-10-01
- 12A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases. Juan Sequeda, Dean Allemang, Bryon Jacob, arXiv, 2023-11-13
- 13Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update. dbt Labs, 2026-04-07
- 14Citations. Anthropic, Claude Developer Platform
- 15LLM06:2025 Excessive Agency. OWASP Gen AI Security Project
- 16Our framework for developing safe and trustworthy agents. Anthropic, 2025-08-04
- 17LLM01:2025 Prompt Injection. OWASP Gen AI Security Project
- 18AI Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology, 2023-01-26
- 19ISO/IEC 42001:2023 Information technology. Artificial intelligence. Management system. ISO, 2023
- 20Regulation (EU) 2024/1689 (Artificial Intelligence Act). EUR-Lex, 2024-06-13
- 21AI Act: regulatory framework for AI. European Commission, 2026-10-19
- 22State of AI trust in 2026: Shifting to the agentic era. McKinsey & Company, 2026-03-25
- 23Artificial Intelligence Strategy of the Republic of Azerbaijan for 2025-2028 (in Azerbaijani). Ministry of Digital Development and Transport of Azerbaijan, 2025-03-19
- 24Presidential Resolution No. PP-358 approving the Strategy for the Development of AI Technologies until 2030 (Russian text). lex.uz, 2024-10-14
- 25Presidential Decree No. UP-189 on additional measures to develop AI technologies (Russian text). lex.uz, 2025-10-22
- 26Government Resolution No. 592 approving the Concept for the Development of Artificial Intelligence for 2024-2029. adilet.zan.kz, 2024-07-24
- 27Law of the Republic of Kazakhstan on Artificial Intelligence, No. 230-VIII. adilet.zan.kz, 2025-11-17
This article is also available in Azərbaycanca, Русский