To curb 'killer' machines in the age of AI

As artificial intelligence increasingly permeates the military sphere, it is evolving into a dominant force in warfare, bringing with it numerous security threats and risks. How to curb "killer machines" is a question that countries must urgently answer together.

The growing use of AI in recent military operations is significantly reshaping modern warfare. AI is incorporated into multiple operational domains, including target identification, intelligence analysis, command and control, and decision-making support.

Besides, it is also embedded in numerous weapons systems, such as loitering munitions (also called suicide drones), and other unmanned vehicles across air, land and sea.

However, with AI systems in play, the entire observe-orient-decide-act (OODA) loop is activated, compressing decision-making time and increasing the risk of unintended outcomes. The rapid pace of warfare reduces the window for commanders to exercise judgment and verification, increasing the likelihood of overreactions in combat. This is further complicated by the unpredictability of the "algorithmic black box", where the internal workings of AI are not fully understood, making it difficult to trace the logic behind its decisions.

As AI assumes a greater role in wartime decision-making, decision times are drastically reduced, increasing the potential for miscalculation, escalation, and loss of control.

When AI generates a target list or operational plan, commanders often see only the output without knowing the underlying rationale, leading to errors in target identification that potentially result in wrongful strikes.

AI is vulnerable to misidentification due to battlefield changes or adversarial tactics such as deepfakes and data poisoning, which can cause it to mistake civilian vehicles for military ones or allies for enemies.

For example, in the Israeli-Palestinian conflict, Israel's "Lavender" system reportedly flagged about 37,000 Palestinian men as suspected armed targets. However, with a 10 percent error rate, this resulted in significant civilian casualties.

Entrusting AI with authority in warfare increases the risk of losing control. Recent conflicts demonstrate that the exponential growth of battlefield sensors overwhelms commanders with data, requiring AI to filter and interpret the situation. This effectively cedes human control to algorithms in determining which intelligence is important.

Left with a sliver of selected information, commanders' situational awareness narrows, exponentially increasing decision-making risks and the likelihood of inadvertent crises.

More importantly, introducing AI into the military threatens nuclear strategic stability. In The Age of AI: And Our Human Future, former United States secretary of state Henry Kissinger warns that AI is eroding the stabilizing mechanisms that have maintained nuclear peace since the Cold War.

On the one hand, AI greatly enhances detection and tracking of nuclear delivery platforms such as strategic nuclear submarines and strategic bombers, fueling incentives for preemptive strikes.

On the other hand, generative AI can fabricate false imagery, communications, and crisis signals such as fake nuclear force movements or missile launches, misleading decision-makers.

In the age of AI, flash points between nuclear states have a lower threshold for escalation. Major countries acknowledge the security risks of military AI applications, but governance has been slow, and significant regulatory challenges persist.

As AI gains strategic importance in military competition, major powers are intensifying R&D and deployment to maintain an edge, and generally would not choose to fall behind.

The formidable military potential of AI has amplified calls for negotiations under the Convention on Certain Conventional Weapons, or CCW, building on years of discussions about lethal autonomous weapons.

However, progress has been blocked under the CCW's consensus rule. Additionally, some nations are using their advantages to impose unilateral standards, shaping a fragmented rules-based order while sidestepping international obligations.

This has fractured global governance consensus and deepened divisions over AI military applications.

Worse still, "lethal and autonomous" AI seems immune to the current arms control regime, which relies on three prerequisites: contestable utility, definable boundaries, and verifiable compliance. The complex nature of AI's military applications eludes all three conditions.

AI's effectiveness in wartime has been demonstrated, prompting countries to rapidly integrate it into their forces. Unlike nuclear weapons, its less destructive nature often bypasses serious deliberation before use.

AI's presence across the entire spectrum, from early warning to strike, creates continuity and ambiguity that challenge precise legal delineation.

Additionally, military AI, embedded in algorithms and software, can be easily disguised as civilian technology, making traditional on-site verification largely ineffective.

Moreover, AI evolves monthly, while international regulations are updated annually, widening the structural gap and creating a persistent governance vacuum and security risks.

The current legal and ethical framework lags far behind, complicating accountability for AI-assisted decisions. Current international humanitarian law and the rules governing warfare are predicated on human intent, foreseeability, and controllable decision-making.

However, "black box" AI algorithms produce unpredictable outputs, making it difficult to assign responsibility for algorithmic misjudgments among developers, deployers, and commanders. The international community has yet to reach a consensus on accountability attribution.

Despite challenges, global regulatory efforts continue. A preliminary consensus on managing military AI risks and prioritizing humanitarian concerns has been reached.

Future efforts should focus on enhancing existing multilateral mechanisms to establish a fair and effective global governance framework for AI's military applications.

It is crucial to ensure that humans maintain ultimate and effective control and prohibit machines from independently making final decisions on weapon use, especially nuclear weapons.

Cold War nuclear control practices by superpowers show that machines must not have the authority to launch strikes. China's principle of human control over AI weapon systems offers guidance for global AI governance.

A practical institutional framework is needed to ensure full-process review, documentation, and accountability for AI's military applications.

This includes clear responsibilities across R&D, deployment, and command-and-control, enabling traceability and accountability for incidents.

China has contributed a position paper under the CCW framework, offering systematic proposals for regulating military AI applications.

It is also important to uphold true multilateralism, harmonize the demands of all parties within the UN framework, and take on board the positions of developing countries.

Strategic trust should be strengthened, and technological risks managed through strategic dialogue, academic exchanges, and military-to-military communication.

AI governance in the military domain should progress inclusively and consultatively.

Li Zhe is an associate researcher at the Academy of Military Sciences, the People's Liberation Army, and Qi Qi is an assistant researcher at the same academy.

The views don't necessarily reflect those of China Daily.

If you have a specific expertise, or would like to share your thought about our stories, then send us your writings at opinion@chinadaily.com.cn, and comment@chinadaily.com.cn.