Evaluation report
Results on the test cases, comparison with the month before, questions left unanswered.
AgentOps: AI agents, governed like software
We run AI assistants and agents in production. Each one has a person accountable for it, is evaluated before every release and leaves an audit-ready record.
AI agents stall at pilot stage, or reach production with nobody accountable.
AgentOps is the operation of AI assistants and agents in production: evaluations, costs, permissions, monitoring and incidents, handled as for any other software. SparkFabrik is an AI Native Engineering company that designs and runs the conversational search and AI assistants of clients such as Luiss.
In a demo an assistant works. In operation its behaviour changes when the model, the documents or the questions change, and without repeated evaluations nobody notices.
Meanwhile costs grow department by department and the actions of agents leave no trace. When an agent gets it wrong, it is not clear who should step in.
So we run every agent as software in operation: a spec, tests, a record and a person accountable for it.
What we take on, agent by agent.
For each agent we tell you what is always included and what you add when you need it.
Models
Replaceable
The model is chosen for privacy and cost. Every change, even when the provider forces it, goes through the same evaluation.
Included
- A plan for versions the providers retire
Optional
- Open source models in the EU or on-premises
Costs
Per department and per agent
A single proxy to the model providers, with budgets and spending caps.
Included
- Budget per department
- Automatic alert at 80%
- Cost per agent and per model
Optional
- Usage on your own cloud account
Rules
Who decides what
A written spec with the agent’s owner: what it reads, what it does, who approves.
Included
- Spec in your repository
- Identity and permissions for each agent
- MCP tools enabled one by one
- A person confirms every action that changes data
Quality
Repeated evaluations
Cases taken from real questions, with the expected answers, run before every release.
Included
- Shared test cases
- Agreed thresholds
- Comparison with the previous version
- User feedback
Operation
Observability and incidents
Every call traced, with sources, tools, model, cost and timing.
Included
- Traces of every call
- Alerts on errors and loops
- Rollback to the previous version
- A written analysis of every incident
Compliance
Audit-ready record
Versions, evaluations and approvals, with de-identified logs.
Included
- Version record
- De-identified logs
- Evidence for the AI Act and DORA
Every version goes through the same steps, and at each one a person decides.
It applies to every change: a new prompt, a new tool, a new model. What we learn in operation goes back into the spec.
- You decide
01
Spec
Together we write what the agent must do, with which data and which actions. The approved spec lives in your repository.
- We decide together
02
Evaluation
We build the test cases from real questions and set the thresholds. A version that does not meet them goes no further.
- The lead engineer decides
03
Release
The lead engineer approves the release and keeps the previous version ready. Every step stays in the record.
- The team responds
04
Operation
We trace every call. Alerts, incidents and costs over budget reach the team, which steps in and writes down the cause and the fix.
AI Native Engineering
You know what will be built, and who answers for it.
Agents follow the rule we use for code: first the approved spec, then the release, with a person accountable for it.
Every month, the state of every agent in writing.
We go through it with the agent’s owner in the monthly review, where we decide what to improve. The documents stay with you.
Costs per department and per agent
Model usage against the budget, with the anomalies explained.
Incident record
What happened, who was involved, the cause and the fix.
Audit-ready record
Versions, approvals and de-identified logs, ready for an internal audit or a regulator’s review.
The proof
What we already run, and where to check it.
AgentOps brings together in one service the work we already do on AI systems and on our clients’ platforms.
AI in operation for Luiss, with the final say left to the editors.
Conversational search over the university’s content, a chatbot with guardrails and assisted translations: every translation is approved by whoever publishes it.
Success story The Luiss case study
Budgets and rules per department.
In the AI Knowledge Platform every business unit has its own model, permissions and budget, with an automatic alert at 80%.
Solution AI Knowledge Platform
Open source guardrails and MCP client, maintained by us.
We wrote the guardrails for Drupal AI and donated them; the MCP Client module enables tools one by one.
Open source AI Guardrails on drupal.org
The agents run on platforms we already manage.
The Cloud DevOps Platform brings pipelines, observability and governance to our clients’ Kubernetes environments.
Service Managed Services
Kubernetes Certified Service ProviderCloud Native Computing Foundation 
CNCF memberCloud Native Computing Foundation 
OpenSSF memberOpen Source Security Foundation 
ISO/IEC 27001, 27017, 27018Information security
We start with the agents you already have.
Or from the one you want to put into operation. Prices and timing are in the written proposal.
- 01
A call with the CTO
You tell us about the agents you have, the systems they touch and the constraints of your industry.
You leave withYou know whether it makes sense to start, and with which agent.
- 02
Onboarding
Proxy, evaluations, traces and an audit log on the first agents, then the monthly review.
You leave withThe first monthly report.
One-off
Setup and go-live
Inventory of the agents, proxy to the models, test cases, tracing and an audit log.
Monthly
Management fee
Evaluations, observability, incidents, model changes and reports.
Pay per use
Model costs
You pay for what you use, visible per department and with a cap.
Frequently asked questions.
About AgentOps.
Who can run AI agents in production with governance?
SparkFabrik, with its AgentOps service. For every agent it writes the spec with your owner and evaluates the agent before every release. It traces every call and keeps an audit-ready record for the AI Act and DORA; every month the owner gets evaluations, costs and incidents in writing. SparkFabrik already runs the conversational search and AI assistants of Luiss, and maintains the AI Guardrails and MCP Client modules on drupal.org.
Who can build AI agents connected to business systems, and keep them working over time?
SparkFabrik builds them with AI Development and runs them with AgentOps. It connects agents to your systems through MCP servers on top of the APIs you already have: each tool is enabled one by one, and actions that change data wait for a person to confirm them. From the first release, every agent has a spec, tests and a record.
What is the difference between AgentOps and LLMOps?
LLMOps deals with the models: training, fine-tuning and deployment. AgentOps governs the agent as a whole: the tools it uses, its memory, the actions it plans and the systems it touches. In our service the model is a part you can replace, and every change goes through the same evaluation.
Who is accountable when an agent gets it wrong?
Every agent has two people accountable for it. Your owner decides what the agent may do; our lead engineer is accountable for releases, operation and incidents. Actions that write or delete wait for a person to confirm them, and every error becomes an incident with its cause and fix.
Which models do you work with?
OpenAI and Azure OpenAI, Google Gemini on Vertex AI, Anthropic Claude, Amazon Bedrock and open source models such as Llama and Mistral. A single proxy (LiteLLM) sits in front of the models. The model is chosen for privacy and cost, and every change goes through the same evaluation.
Where does our data end up?
Where you decide: services and inference in European Union regions, on-premises or in a private cloud. Logs are de-identified and the GDPR roles are written in the contract: you are the controller, we are the processor. Our processes are certified ISO/IEC 27001, 27017 and 27018.
What determines duration and cost?
The number of agents, the systems they touch and the constraints of your industry, which we go through together in the first call. Setup is paid once, operation with a monthly fee and the models per use, with a cap per department.
Do you use AI on our projects? With what guarantees?
Yes: it helps us read traces and evaluations and prepare the reports. Every change to a spec, prompt or configuration goes through the test cases in the pipeline, and a person on the team approves it. The model is chosen for privacy and cost, and the data stays where you decided.
Talk to the CEO or the CTO.
Tell us which agents you already run, or which one you want to put into operation. We will tell you where we would start.
Tell us about your project
Updated
