AI privacy inference can expose personal facts from writing that never states them outright. In a peer-reviewed ICLR 2024 study, researchers at ETH Zürich tested large language models on real Reddit profiles and found that GPT-4 reached about 85% top-1 accuracy and 95.8% top-3 accuracy when inferring attributes such as location, income and sex from text. The system was not simply searching for a name, postcode or salary figure. It was reasoning from clues.

That changes the privacy problem. Most people know not to post an address, phone number or date of birth publicly. Far fewer expect an AI to combine ordinary details, habits, cultural references and writing patterns into facts they deliberately left unstated. The researchers found that the process could also be more than 100 times cheaper and more than 240 times faster than comparable human profiling, turning something once labour-intensive into something potentially scalable.

AI Privacy Inference Means Anonymous Text May Not Be Anonymous

The study, Beyond Memorization: Violating Privacy via Inference with Large Language Models, was presented at the International Conference on Learning Representations in 2024. The researchers constructed a dataset of 520 real Reddit profiles containing thousands of comments and manually annotated personal attributes including age, education, sex, occupation, relationship status, location, birthplace and income.

They then asked several language models, including GPT-4, Claude 2, PaLM 2 and the Llama 2 family, to infer those attributes from the users’ comments. According to the researchers’ project page, GPT-4 achieved an average top-1 accuracy of 85% and top-3 accuracy of 95.8% across the task. The distinction matters: top-1 means the model’s first answer was correct, while top-3 means the correct answer appeared among its three best guesses.

The models were not being tested on whether they could retrieve private records from their training data. The comments were provided to them at inference time. The privacy risk therefore came from reasoning over what the user had written, not from reproducing a memorised database entry. That is why the researchers called the problem “beyond memorization”.

What This Means for You: Removing Obvious Identifiers Is Not Enough

Imagine posting online under a pseudonym. You never state where you live. You never give your age. You avoid naming your employer. Conventional privacy advice might suggest that you have removed the important identifying information. AI privacy inference attacks a different layer: the information hidden in context.

A later paper from the same ETH Zürich team gives a useful example. A short post mentioned the “left shark” incident, walking nearby to see it after final exams, but contained no direct location. The researchers explained that contextual knowledge could connect the reference to the 2015 Super Bowl halftime show and infer Glendale, Arizona. A traditional entity-removal tool had nothing obvious to redact because the place name was never present in the text.

This is the central psychological trap. Humans tend to think about privacy in terms of facts they consciously disclose. AI can instead treat language as a collection of weak signals. One comment might reveal almost nothing. Ten comments about transport, school systems, food, weather, television, working hours and local customs can form a much richer profile when combined.

That makes the problem quite different from the one covered in LiveAIWire’s guide to what happens when you delete an AI chat. Deletion asks what happens to information you knowingly supplied. Inference asks what additional information can be produced from what you supplied, even if you never typed the private fact itself.

How AI Works Out Facts You Never Actually Wrote

Large language models are unusually suited to this task because they combine broad world knowledge with contextual reasoning. A sentence about taking a particular tram, celebrating an obscure local tradition, using a regional phrase or sitting a particular type of school examination may each narrow the range of plausible locations or demographics.

The ETH researchers demonstrate the idea through an interactive privacy inference game. One example describes a visitor arriving in a country surrounded by the Alps, taking tram 10 for eight minutes from the airport towards an arena and hoping to try famous cheese. No city is named, but the clues can be combined into a location hypothesis. The important feature is not any single clue. It is the model’s ability to join them.

The research also found that larger, more capable models generally performed better at private-attribute inference. That does not prove every newer model will always infer private facts more accurately. It does show that privacy can become an unintended side effect of improved general reasoning. A model does not need to be designed as a surveillance tool to possess capabilities that can be repurposed for profiling.

This resembles a broader tension in AI systems: capabilities developed for useful tasks can have secondary uses their designers did not optimise for. LiveAIWire has examined a similar pattern in AI deepfakes in everyday life, where tools built to generate realistic media can also support impersonation and fraud.

The Scale Is What Makes the Research Uncomfortable

A skilled investigator has always been able to infer things from language. People recognise accents, slang, cultural references and occupational jargon. The important change is automation. The ETH team reported that LLM-based inference could reduce the monetary cost by more than 100 times and the time required by more than 240 times compared with human profiling in their evaluation.

That difference changes who can be profiled and at what scale. Manually examining thousands of pseudonymous users is expensive. Passing thousands of public comments through an automated system is much easier. The research therefore points to a privacy risk that is not necessarily new in kind, but potentially new in volume.

Scale also makes mistakes more consequential. An AI profile can be wrong while still being acted upon. If a system incorrectly infers someone’s income, age, politics or location and that inference feeds into advertising, risk scoring, moderation or targeting, the person may never know the hidden assumption existed. Accuracy does not need to reach 100% for the process to affect people.

That is why this topic connects to LiveAIWire’s coverage of AI systems making financial decisions. In high-stakes systems, inferred characteristics can matter even when the person concerned never explicitly supplied them. The difficulty is not merely whether an algorithm knows something. It is whether an invisible inference becomes an input into another decision.

Why Ordinary Anonymisation Can Fail

The obvious defence is to anonymise the text. Remove names, addresses, email addresses, phone numbers and other personally identifiable information before publishing or processing it. The ETH researchers tested that idea and found a serious limitation: conventional anonymisation tools can remove explicit entities while leaving contextual clues intact.

In a follow-up study, Large Language Models are Advanced Anonymizers, the team evaluated 13 language models and compared a new adversarial anonymisation approach with existing methods. Their argument was straightforward. Traditional systems are good at detecting strings that look like names, locations, identification numbers or email addresses. They are much less effective when privacy leaks through meaning rather than through a recognisable entity.

The “left shark” example makes the problem visible. There is no word “Glendale” to remove. Protecting the location requires understanding why the event reference, timing and surrounding context might reveal it. The researchers therefore used one language model to infer sensitive attributes and another to rewrite the text to frustrate those inferences while trying to preserve its meaning.

The follow-up evaluation found that this adversarial LLM-based anonymisation provided a better privacy-utility trade-off than the comparison anonymisers, and a human study with 50 participants showed a strong preference for the LLM-anonymised text. That is encouraging, but it also produces an odd arms race: increasingly capable AI may be needed to defend writing from increasingly capable AI inference.

Even Your Photos Can Leak Facts You Never Typed

The privacy issue is no longer confined to text. Researchers including members of the same ETH group later tested private attribute inference from images with vision-language models. They deliberately used images where private attributes were not simply visible on a person’s face or body. Across seven models, the best results reached up to 77.6% accuracy on some private-attribute inference tasks.

The implication is broader than one benchmark. A kitchen, street scene, interior, meal, landscape or collection of objects may carry contextual information about location, lifestyle or socioeconomic circumstances. As multimodal systems improve, text, images and other signals can potentially be combined rather than assessed separately.

This does not mean every holiday photo gives an AI an accurate dossier about its owner. The reported results come from constructed evaluation datasets and vary by attribute and model. But they reinforce the same principle seen in the Reddit study: the absence of an explicit private fact does not mean the input contains no evidence about that fact.

People Are Not Necessarily Most Worried When the AI Is Correct

A 2026 preprint adds an interesting human dimension. Researchers studying how users react to personal inferences made from their own ChatGPT histories showed 18 regular users 215 example inferences. Participants often responded with curiosity rather than alarm. Discomfort was stronger when an inference felt inaccurate, misrepresentative or likely to be used outside the context the user expected.

The study was small and should not be treated as a population-wide measure of attitudes. Its value is conceptual. Privacy is not only about whether an inference is sensitive. It is also about who generated it, why, whether it is retained, and who else can use it. A user may welcome an AI inferring that they prefer concise answers while strongly objecting to an advertiser inferring income, health status or political beliefs from the same conversation.

That contextual distinction matters as AI products become more personalised. A system that understands more about a user can be more useful. The same inferential ability can also become a profiling mechanism. Personalisation and surveillance are not identical, but technically they can rely on overlapping capabilities.

Deleting the Data Does Not Necessarily Delete the Inference

This creates another complication for privacy rights. LiveAIWire has previously examined the AI right to be forgotten, where the central problem is removing personal information from systems that may already have processed it. Inferred information adds a separate question: what happens when the system can recreate a private attribute from data that remains lawful to hold?

Suppose a platform does not store a field saying “income: £70,000”. Instead, a model estimates an income range whenever it analyses a person’s posts. Deleting a stored income field would solve nothing because there may be no such field. The sensitive result is generated dynamically from other information.

This does not mean privacy law provides no protection for inferred data. The legal treatment depends on jurisdiction, purpose and whether information relates to an identifiable person. The technical point is simpler: systems designed around finding and deleting explicit data fields may not address information that can be regenerated from context.

The Best Defence Is to Think in Clues, Not Just Secrets

There is no perfect practical rule for preventing AI privacy inference while continuing to communicate normally online. Language works because it contains context. Strip away every clue about where you live, what you do and how you live, and much of what makes a post useful or human disappears as well.

The more realistic approach is to recognise that pseudonymity and anonymisation are not guarantees. When posting highly sensitive material, consider not only the facts you state directly but combinations of contextual details that could narrow your identity or circumstances. Separate accounts can reduce linkability, but repeated writing across accounts can still accumulate clues. For confidential material, processing data locally can also reduce who receives the raw input, which is one reason on-device AI has a genuine privacy advantage over cloud processing.

None of these precautions makes inference impossible. The research instead argues for better technical defences, clearer controls over how inferred information is used and a wider definition of what privacy protection needs to consider.

AI Privacy Inference Changes What It Means to Reveal Something Online

The unsettling part of this research is not that AI can read a postcode. Software has been able to do that for decades. It is that a model can sometimes reconstruct private attributes from fragments that look harmless in isolation.

That changes the mental model ordinary users need. Privacy is no longer only about protecting the secret itself. It is also about protecting the clues from which the secret can be derived.

The ICLR research showed this at meaningful scale using models available in 2023 and 2024, with GPT-4 reaching about 85% top-1 accuracy across the researchers’ private-attribute task. Follow-up work has since explored stronger anonymisation, inference from images and how users feel when unstated conclusions are surfaced from their own conversations.

The deeper question is therefore not whether you told an AI your age, income or location. Increasingly, it may be whether you told it enough other things for it to work them out.

About the Author

Stuart Kerr is Technology Correspondent at LiveAIWire, covering artificial intelligence, cybersecurity, and the social impact of emerging technology. He publishes daily at LiveAIWire.com.