Cisco's AI Threat Intelligence and Security Research team said that standard AI safety benchmarks significantly understate the threat of AI-driven cyber attackers [1].

This finding suggests a critical gap in how the industry measures risk. If current benchmarks fail to capture the true capabilities of AI-enabled threats, organizations may be underprepared for sophisticated attacks that bypass existing safety guardrails.

The research team said that emerging AI-driven attackers are already becoming a reality [1]. According to the report, the existing frameworks used to test AI safety are insufficient because they miss key risk factors [1]. This oversight leads to a general underestimation of what AI-enabled attack capabilities can actually achieve in a real-world environment.

Security professionals often rely on these benchmarks to determine if a model is safe for deployment or if it possesses "dangerous capabilities." However, Cisco's findings indicate that the metrics currently in use do not align with the actual tactics employed by malicious actors [1].

The team said that the emergence of these attackers is not a distant possibility but a current development [1]. By failing to account for these specific vectors, the industry risks a false sense of security regarding the robustness of AI defenses.

Cisco's global research team said that the disconnect between benchmark results and real-world threats creates a vulnerability in global cybersecurity infrastructure [1]. The report suggests a need for more comprehensive testing that reflects the evolving nature of AI-driven exploitation.

“Standard AI safety benchmarks significantly understate the real threat.”

The discrepancy between AI safety benchmarks and actual threat capabilities indicates that the industry's 'safety' metrics may be lagging behind the offensive innovation of cybercriminals. This shift suggests that defensive strategies must move beyond static benchmarks toward dynamic, threat-informed testing to prevent large-scale exploitation by AI-powered actors.