AI Agents Need More Than Capability

When companies test AI agents, the easiest metrics to measure are accuracy, task completion, tool-use success, speed, and cost.

Those metrics matter because they answer a basic question:

But once an agent moves beyond answering questions and begins connecting to CRM systems, knowledge bases, ticketing platforms, or other enterprise tools, the deployment question changes.

If the agent can execute actions after receiving authorization, proving that it can perform the task is no longer sufficient.

The enterprise must also be able to answer:

That creates a second readiness threshold as AI agents move from proof of concept into production.

QwenWork offers a concrete example

Alibaba's enterprise AI platform QwenWork illustrates how this control layer is beginning to appear in commercial agent platforms.

According to its product documentation, authorized agents can use connectors to retrieve information from external systems and perform operations supported by those services.

Enterprise administrators can manage which connectors are available to the organization, review connectors submitted by employees, enable or disable them, and manage related authorization.

That matters because the agent is crossing an important boundary.

It is no longer limited to generating an answer. It may be able to take action inside a business system.

Once that happens, model quality is no longer the only production-readiness issue.

Enterprises may need two separate deployment tests

A practical approach is to separate production readiness into two categories.

Capability Test

A capability test asks:

An enterprise might measure:

• Task success rate

• Accuracy

• Tool-use capability

• Execution speed

• Cost

• Work quality

These remain necessary.

But they demonstrate that the agent is capable of doing the job. They do not, by themselves, demonstrate that the agent should be given execution authority in a production environment.

Control Test

A control test asks:

Based on the QwenWork capabilities that can currently be verified, that question can be broken into at least five areas.

1. What can the agent access?

The first issue is access scope.

Which data, accounts, connectors, and enterprise systems can the agent reach?

If an agent only needs inventory information, for example, does it really need broad access to a CRM system or other customer data?

Enterprises need to determine whether access follows a least-privilege model rather than granting broad system access simply because the technology makes it possible.

2. What is the agent authorized to do?

The second issue is execution authority.

Reading data and changing data represent different levels of risk.

Looking up customer information, creating a ticket, modifying an order, sending an email, deleting data, or approving a transaction should not automatically receive the same level of authority simply because the agent is technically capable of performing those actions.

An enterprise needs to define:

• Which actions can the agent execute autonomously?

• Which require additional controls?

• Which should not be delegated to an agent at all?

3. Can an unsafe action be stopped before it happens?

This is one area where QwenWork’s technical documentation provides relatively clear evidence.

QwenWork Enterprise Flagship supports a PreToolUse mechanism that allows an enterprise to inspect the tool and parameters before the agent actually invokes it.

The enterprise can then decide whether the action should be allowed or blocked.

Its documentation specifically indicates that organizations seeking to prevent writes, sends, or deletions should perform that check before execution.

That distinction matters.

For an enterprise, one of the most valuable controls may not be the ability to investigate a bad action afterward.

It may be the ability to prevent the action from happening in the first place.

4. If something goes wrong, can the enterprise determine what happened?

The fourth issue is traceability.

QwenWork provides AI logs and administrative operation records that allow enterprises to inspect agent use, related resources, and changes to management settings.

That gives organizations some ability to reconstruct activity after the fact.

There is, however, an important limitation.

Current documentation indicates that the basic unit of the AI log is a conversation turn, and a single turn may contain multiple resource calls.

The available evidence therefore does not establish that every individual tool action has a complete, independent, action-level audit record.

For many enterprises, the available logging may still be operationally useful.

For regulated industries such as financial services or healthcare, however, whether that level of granularity satisfies formal audit requirements would need to be verified separately.

5. Can a completed action actually be reversed?

This is one of the easiest capabilities to overestimate.

QwenWork product materials have referred to protection against mistaken deletion and recovery capabilities.

Its technical documentation also makes clear, however, that once PostToolUse occurs, the tool operation has already been executed and cannot be undone through that mechanism.

That means:

If an agent has already changed something in a third-party SaaS application, whether that change can be reversed depends on the workflow, the external tool, and the underlying system.

Recovery therefore needs to be tested action by action.

Enterprises should not treat “recovery support” as equivalent to universal rollback.

A successful proof of concept demonstrates capability, not necessarily production readiness

This is the most important decision insight from the case.

A strong PoC result demonstrates:

Production deployment requires another body of evidence:

Those are different questions.

That distinction helps explain why an agent that performs well in tests may still not be ready to update a CRM record, process a contract, send an email, modify a ticket, or take another action inside a production system.

Moving from PoC to production may therefore require more than improving the same performance metrics.

It may require a second set of control evidence.

Agent procurement criteria may need to change as well

If that distinction holds, enterprise procurement and vendor evaluation may also need to evolve.

Beyond accuracy, speed, cost, and task completion, buyers may need to ask:

• What data can the agent access?
• How granular are its permissions?
• Which actions can be intercepted before execution?
• Which actions require human approval?
• Can access be revoked immediately?
• At what level can activity be audited?
• Which completed actions can actually be recovered?
• Which external actions become irreversible once executed?

Those questions help determine whether an agent remains an experimental tool or is ready to become part of a production operating environment.

QwenWork does not prove that enterprise agents have solved the control problem

The evidence should not be extended beyond what it supports.

Public documentation confirms that QwenWork has implemented several enterprise control capabilities, including organization-level connector management, authorization revocation, pre-execution blocking, and activity logging.

The available evidence does not establish that:

• Every high-risk agent action is subject to a standardized human approval process.
• Every running task can be stopped through a universal real-time kill switch.
• Every completed external action can be rolled back.
• These controls have reduced real-world agent incident rates.
• Enterprise customers have achieved independently verified ROI because of them.
• Chinese enterprises have broadly moved from agent PoCs to scaled autonomous execution.

QwenWork is useful here as a concrete evidence case.

It is not proof that the broader market has reached production maturity.

Today's Decision Insight

The next maturity question for enterprise AI agents may be shifting from:

to:

For companies already running agent pilots, one useful way to rethink the production gate is:

Capability readiness demonstrates that the agent can perform the task.

Control readiness demonstrates that the organization still controls:

• What the agent can access
• What authority it has
• Which actions can be stopped before execution
• What can be reconstructed after an action occurs
• Which mistakes can actually be recovered

The value of this distinction is that it turns the question of whether an agent is “ready” into a set of operational conditions that can be tested.

When AI only provides recommendations, model quality may remain the central issue.

When AI begins to receive execution authority, whether the enterprise still retains control becomes a separate production-readiness requirement.

This Issue's Decision Question

Turning Global AI Signals into Business Decisions

InfoAI helps decision-makers identify what is changing in AI, evaluate the evidence behind those changes, understand what they mean, and determine which business assumptions or decisions may need to be reconsidered.

www.InfoAI.com.tw