In AI Battleship tests, researchers found that a relatively small AI model became a much more effective player in a version of Battleship when it learnt to ask more useful questions. In the AI Battleship study described by MIT in June 2026, the computer was not simply given a bigger collection of facts. Instead, it was given a way to weigh possible hidden ships, choose questions that reduce uncertainty and decide when it had enough information to make a move.
The findings challenge an easy assumption about powerful AI: that the biggest model will always make the smartest decision. In this carefully defined game, researchers reported that Llama 4 Scout, a smaller language model, improved from beating human players in eight per cent of games to 82 per cent when equipped with a structured reasoning method. That is a striking laboratory result, but it must not be presented as a general claim that a small chatbot is now better than every large one.
The game is interesting because it isolates a skill that people use constantly. Before making a costly choice, a sensible person looks for missing information. The next question often matters more than the speed at which somebody answers the previous one. The researchers wanted to test whether AI systems could do the same.
Why AI Battleship rewards better questions
Ordinary Battleship is a game of hidden information. A player does not know where the opponent’s ships are. Every move gives a little evidence, which can either strengthen or rule out possible positions. A good player is not choosing squares at random. They are trying to make each turn reveal something useful, while still taking opportunities to hit ships when enough clues are available.
MIT and Harvard researchers created a variation called Collaborative Battleship. One participant, the captain, asks questions and decides where to act. Another, the spotter, knows the hidden board and supplies answers. The design lets researchers test two separate abilities: whether a questioner seeks valuable information and whether the other player responds accurately. A system can fail at either job, even if it writes natural-sounding sentences.
The researchers first gathered examples from 42 human players. They then compared those patterns with the behaviour of language models, including larger frontier models and smaller alternatives. Some AI systems could complete games quickly, while others asked questions that did not reveal much. The weakness was not always a lack of factual knowledge. It was a lack of discipline about what would be useful to know next.
That distinction has a familiar everyday equivalent. Somebody helping you solve a computer problem might immediately recommend reinstalling software. A better assistant would first ask what changed, whether the problem affects one application and what the screen actually shows. Those questions can narrow the possibilities before anyone risks losing data or making the fault harder to understand.
The trick was to keep several possible answers in mind
The team used a method for considering multiple possible hidden boards as new clues arrived. Rather than settling too early on one supposed answer, the computer maintained and updated plausible arrangements. It then asked questions expected to reduce uncertainty or guide the next useful move. Researchers call the general approach probabilistic inference, but the everyday idea is simpler: imagine several explanations, test them and adjust your beliefs when new evidence appears.
That is not the same as making a language model conscious of its own doubts. The system uses a designed procedure to track possibilities. Its improvement comes from linking the language model to a structure for making decisions, not from assuming the model has acquired human judgement. This is an important distinction when the result is described outside its research setting.
The research paper, first posted in October 2025 and revised in March 2026, provides the underlying comparisons. In its reported evaluations, the refined smaller model outperformed human baselines on the game and also performed better than GPT-5 on particular comparisons at around one per cent of the stated model cost. The denominator matters: this is a cost comparison for the researchers’ defined task, not a universal price comparison across all possible work.
It is also worth separating the game result from real-world success. A controlled board has rules that can be encoded, a finite set of possibilities and clear success criteria. Open-ended tasks in a workplace may have conflicting objectives, missing records and no single correct answer. The method is promising because it makes questions more deliberate, but transferring it beyond a board game requires evidence from new tests.
When the person answering gets the board wrong
The spotter’s role revealed another weakness. Even when a language model has access to relevant information, its answer can be unreliable if the question requires careful checking. A natural sentence that sounds confident is not necessarily tied to the actual position of the pieces. The researchers found that this problem affected models of different sizes, though the extent varied.
To improve reliability, they translated questions into explicit computer instructions that could inspect the game board. A question about whether a ship occupied a particular region became something the software could check against the actual arrangement. Across the tested systems, the approach improved answer accuracy, with the paper reporting gains of up to 14.7 percentage points over relevant baselines.
This is a form of verification rather than persuasion. The computer is not asked to sound more certain. It is given a test that can establish whether the answer matches a known state. That matters beyond games because many everyday AI errors occur when a fluent explanation is mistaken for a checked result. A user may need the actual figure in a database, the correct item on an invoice or the exact clause in a document.
LiveAIWire has explored the need to verify AI-supported mathematical claims. The question-answering method used in this research offers a related lesson: when there is a reliable external way to check something, the AI should be encouraged to use it. A verbal argument that an answer is correct is not a substitute for a test against the relevant facts.
A board game can reveal a serious design problem
Research games can be valuable because their simplicity exposes behaviour that is difficult to separate in complicated software. If an agent fails in a business workflow, the cause might be a missing permission, a broken connection, an unclear request or poor reasoning. In Collaborative Battleship, the researchers can observe where questions are unhelpful and when an action is taken too early. The test is small enough to study, yet still demands judgement under uncertainty.
The team also evaluated similar methods in a version of Guess Who?, another game in which the point is to rule out hidden possibilities by asking discriminating questions. That additional test matters because it suggests the improvement was not exclusively tied to the placement of ships on a board. It does not, however, establish effectiveness across every category of reasoning or complex professional decision.
LiveAIWire has also examined AI playing Stratego, another hidden-information game. Such stories are related because success depends on acting without full knowledge of an opponent’s position or intentions. But winning at games and making trustworthy decisions about people’s lives are different standards. A useful research result should not be promoted to a claim about safe deployment without further evidence.
Researchers discuss possible relevance to coding, scientific discovery and other tasks where the best next step depends on asking a smart question. Those applications are reasonable avenues for investigation, rather than demonstrated outcomes of the Battleship experiments. The right follow-up is to test whether the method improves real decisions when information is incomplete and mistakes have genuine costs.
Why a smaller model could be attractive
Large AI models are often costly to run, especially if software makes repeated calls during a multi-step task. If a smaller system can solve a problem by using a clearer decision procedure, a developer may obtain useful performance without relying entirely on greater model size. This could make certain agents cheaper or faster, though any practical saving depends on the additional processing, tools and checks the system needs.
The Battleship experiment illustrates the trade-off neatly. The researchers gave smaller models a framework to examine possibilities and choose better questions. Their comparisons showed substantial gains on the defined tasks. In another environment, the same framework might be too slow, too difficult to maintain or dependent on having a clear representation of all possible outcomes. There is no guaranteed saving until the complete system is measured.
Another question is how to report confidence honestly. A system may be able to keep track of possible answers internally, yet still give users a single conclusion without explaining the alternatives. A trustworthy assistant should make material uncertainty visible when it affects a decision. That does not require publishing pages of technical reasoning. It can be as simple as saying what evidence is missing and what would change the answer.
Research into AI agents using memory connects with that problem. Remembering an earlier fact is only useful if the system can decide whether the fact is relevant now. Likewise, asking a new question is only valuable if it distinguishes between possibilities that matter to the next action. Good information management is about selection as much as storage.
The next useful question is the real takeaway
The most compelling result from the research is not a trophy for a chatbot playing a board game. It is a demonstration that structured question-asking can materially improve performance, including for a model that was initially relatively weak. The researchers brought together human examples, a controlled game and a clear way to test the quality of each move. That combination helps explain why the reported gains occurred.
It would be premature to translate the result into promises about professional diagnosis, independent scientific discovery or dependable financial advice. Those fields introduce evidence quality, ethics, accountability and consequences that a game cannot reproduce. Their agents need more than a clever method for narrowing possibilities. They need reliable tools, oversight and clear limits on what decisions software is permitted to make.
The lesson for anyone building an AI assistant is nonetheless practical. Before producing an answer, the system should sometimes work out which question would reveal the most useful missing fact. In this study, that change made a smaller model much more competitive. Outside the laboratory, the challenge is to make such questions accurate, relevant and worth asking before somebody acts on the reply.
About the Author
Stuart Kerr is Technology Correspondent at LiveAIWire, covering artificial intelligence, cybersecurity and the social impact of emerging technology. LiveAIWire is an independent, human-led technology publication using AI-assisted research, editorial production and original AI-assisted editorial illustrations under his direction.