When Pokémon GO first appeared, most people saw it as a fun mobile game. We caught Pikachu on our screens and walked a little more in the streets. It turns out we were also collecting street imagery that could eventually be useful to delivery robots. The game’s developer, Niantic, now states explicitly in its privacy policy that it collects location information, device identifiers, usage data, and—in some circumstances—visual data. The company also announced in 2024 that it had reached 10 million scanned locations worldwide and that those scans formed a major input for its spatial mapping system. In other words, while millions of people were playing a game, they were simultaneously helping build a machine-readable layer of the physical world.

Today, generative AI tools likewise appear to be intelligent conversational partners and sources of information. Yet these systems can access documents we upload and process our files. Some products work with browser context; others connect to calendars, email, meetings, and corporate content or organize directories on our computers. These systems are becoming capable of constructing increasingly detailed behavioral and contextual profiles. Are you ready to be manipulated by computers?

The issue is not limited to whether you have enabled permission for your data to be used in model training. The fact that data do not enter the training of a foundation model does not make them harmless or risk-free. They may be retained, logged, reviewed by humans, used to improve a product, or made available for profiling.

What kind of representation of you do these systems construct? Which habits of thought, decision-making patterns, vulnerabilities, and priorities are they recording more systematically than you do yourself? This goes beyond the traditional debate on privacy. It raises the prospect of archiving behavior in ways reminiscent of the Cambridge Analytica scandal.

In their official statements, OpenAI says that business users’ data are not used to train models by default. Microsoft 365 Copilot similarly states that prompts and responses are not used to train the foundation models. Google has described an 18-month default retention regime for Gemini Apps Activity and human review under certain conditions. Corporate AI customers may receive stronger protections, while individual users operate in a looser and harsher marketplace for data. Digital inequality begins with the consumer.

Perhaps, as with Pokémon GO, within less than a decade we will begin experiencing the consequences of the systems to which we have opened the details of our daily lives. We therefore need to discuss the politics of data. The conditions imposed by default settings must be clear and understandable. Model training, data retention, and contextual monitoring require explicit regulation. Users must have a meaningful right to refuse. Otherwise, AI risks changing from a tool that helps us into a system that continually analyzes us and may manipulate us at will—assuming it has not already started.