Customers Lack Confidence in Agents for Security; Data Reinforces AI’s Need for Human Oversight

---

New data from Microsoft and MIT Technology Review provides important insights on customers’ confidence level in AI agents to perform security functions — where there’s significant room for improvement — and their belief that humans must retain strong AI oversight.

This second analysis breaking down key findings from the Microsoft-MIT data will provide insights into several of the security-oriented tasks where customers report low confidence levels, as well as chief areas of AI concern including accountability for decision making and inaccurate results.

I’m rating any reported confidence level below 50 (on a 100-point scale) to be low with significant room for improvement – and security-oriented functions are prominent in this category.

The better news: there are 28 tasks (of a total 101 measured) where customers report confidence in AI agents at higher than 70, and I addressed many of these in the first analysis.

Security is an especially important use case because experts recognize the need to use AI tools – and agents specifically – to help combat offensive AI measures aimed at identifying and exploiting vulnerabilities. The concerns about AI are one useful indicator of how and where companies will deploy humans – at least for the foreseeable future — to oversee the work of AI.   

Low Security Confidence

The research shows 14 tasks for which respondents have confidence less than 50 (out of 100) in AI agents. I would classify at least half of those as security functions, including:

• Disaster recovery testing and validation – 43/100 confidence
• Change management risk assessment – 48
• Network intrusion detection pattern analysis – 48
• Memory leak detection and troubleshooting – 48.5
• Security policy enforcement and deviation detection – 48.5
• Cloud security posture assessment – 49
• Compliance monitoring for data regulations – 49

It would be difficult to overstate the need for – and the value of – deploying AI to combat automated attacks; customers and the vendor community recognize that need, as detailed in these reports:

• Cybersecurity Report Underscores the AI Imperative in Combating Automated Attacks
• Security Leaders Ramp Up Agentic Use Cases While Protecting Against Shadow AI

While the urgency is acknowledged, it’s clear that customers don’t yet have high confidence that agentic AI is the solution to their security challenges; that fact points to a significant opportunity for vendors and partners alike to build out AI-powered security capabilities and deliver expertise and services to tap AI for the expanding scale and scope of security threats in the AI era.

AI Concerns – and Mitigation Measures

The research clearly indicates AI must continue maturing in terms of reliability and accuracy. Here are a few areas worth highlighting among concerns raised by respondents teams about AI:

• Overall, 48% of respondents (the highest percent) rated accountability for decision making as a concern; 54% of executives and 52% of team leads raised that concern.
• 47% overall cited inaccurate results and hallucinations as risks; when drilling deeper, 46% of executives and 41% of team leads cited that risk.
• Cost and resource consumption, surprisingly, were raised as concerns by only 22% of respondents. The breakdown in this case is highly variable: 43% of executives are concerned about cost and resource consumption, compared to 17% of team leads, and just 9% of individual contributors.

On the last point, team leads and individuals should actually be more concerned about cost considerations. But cost management viewpoints are evolving rapidly.  (There’s an excellent breakdown of the costs – in practice — of the popular Microsoft Copilot Cowork tool from CIO Kenny Mulllican that I recommend you review for more insight.)

When it comes to ways that customers can mitigate their AI concerns, the leading approach, cited by 59% of respondents, is to keep humans in the loop, followed by increasing observability, cited by 53%. A powerful illustration of these human oversight requirements was delivered in recent weeks by Ford, which acknowledged it went too far, too fast with AI replacing humans in critical business functions, and eventually reverted (with strong results) to relying more heavily on humans with AI support.

Concluding Thoughts

The Microsoft-MIT research makes a strong contribution to the ongoing discussion around use of AI to combat the security threats resulting from AI including automated vulnerability detection. It also provides valuable reminders that humans remain necessary – albeit in new and evolving roles. The relatively low focus on cost considerations is the biggest surprise; it seems out of step with today’s strong customer focus on virtually all cost-related AI factors.