Most large AI systems have used some form of user data in training, fine-tuning, or evaluation, although the exact mix varies wildly by provider and product tier. Standards like the EU's GDPR give some users a real opt-out path; vendor controls such as OpenAI's data settings give others a partial one. What you can actually do right now: check your account's data-sharing settings, review your website's crawler permissions, and push sensitive work into enterprise or privacy-first tiers before you type it anywhere else.
TL;DR:
- Most training data comes from web scraping, user content, licensed datasets, and data broker feeds, with web scraping being the largest and least transparent source.
- Turning off data-sharing settings prevents future data from being used but does not remove influence from past training, which is embedded in model weights.
- Privacy-preserving techniques like differential privacy, federated learning, and inference-time embeddings can reduce risks but often involve trade-offs in accuracy or technical complexity.
- Opting out of training data collection largely limits future input but cannot undo the influence of data already incorporated during earlier training phases.
- Individuals can reduce exposure by adjusting account settings, avoiding sharing sensitive info online, and moving sensitive work to enterprise or privacy-focused tiers.
Table of Contents
- How AI training uses user data
- Where does training data actually come from?
- Can you actually opt out of AI training?
- What privacy-preserving techniques actually work?
- What risks come from training AI on user data?
- How can individuals reduce their exposure?
- How should organisations govern AI training data?
- What should you demand from AI vendors on transparency?
- How Seven approaches user data in project management
- What's the honest trade-off here?
- Sources
- FAQ
How AI training uses user data
Training an AI model isn't one event. It's usually three separate stages, and each one touches user data differently.
Pretraining builds the foundation model on enormous, mostly public datasets scraped from the web, books, and licensed sources. This is where a model learns grammar, facts, and general reasoning. Individual user data rarely dominates here, but it can still be present if a company folds prior user interactions into that corpus.
Fine-tuning is where personal data gets more directly involved. Companies take a pretrained model and adjust it using narrower, often user-sourced datasets, such as customer support transcripts or chat logs, to sharpen performance on specific tasks. Reinforcement learning from human feedback (RLHF) adds another layer: human reviewers rate model outputs, and those ratings, sometimes tied to real conversations, shape future behaviour.
Evaluation tests a model's performance, and evaluation sets sometimes reuse the same user-submitted content collected during earlier stages.
- Consumer chatbot conversations are commonly reviewed and used to improve future model versions unless a user opts out.
- Labelled datasets for tasks like sentiment analysis or content moderation frequently draw on real user-generated posts and messages.
- Customer support logs get repurposed to train response-suggestion or summarisation features.
- Uploaded files and documents in some consumer apps become training material under the platform's default settings.
Here's the part people miss: deleting your account doesn't undo any of this. Once your data has shaped a model's weights during training, that influence is baked in mathematically, not stored as a retrievable file you can request removed. Researchers call the process of trying to strip that influence back out "unlearning," and it remains an active area of study rather than a solved problem. A deletion request stops future use. It doesn't reverse past use.
Where does training data actually come from?
Four main pipelines feed most AI training datasets, and each carries a different privacy profile.
Public web scraping is the largest source for foundation models. Crawlers sweep websites, forums, and social media platforms, and the crawler's user-agent string (like GPTBot) is often the only signal a site owner gets about who's collecting their content. Social platforms scrape their own user base too, sometimes for the platform's own model and sometimes for licensing deals with outside AI companies.
User-submitted content inside apps is a second, more direct channel. Free-tier consumer tools often default to using your prompts, uploads, and conversations for model improvement unless you dig into settings and turn it off. Paid or enterprise tiers frequently carve this out entirely as a contractual term, which is one reason business plans cost more.
Licensed datasets come from publishers, research institutions, or companies that sell structured content collections. These are cleaner from a provenance standpoint but can still contain personal information if the underlying content included user comments, reviews, or forum posts.
Data broker feeds are the least visible source. Brokers aggregate records from multiple platforms and sell them onward, and a person's data can end up in a training set without that person ever interacting with the AI company directly.
- Web and social scraping: broad coverage, weak individual consent trail.
- In-app user content: direct collection, often covered by terms of service most people never read.
- Licensed datasets: contractual and documented, but not always scrubbed of embedded personal data.
- Data broker feeds: aggregated and resold, with the thinnest transparency of the four.
Can you actually opt out of AI training?
Partly, and the practical answer depends heavily on where you live. In-app privacy portals typically expose a toggle, often labelled something like "improve the model" or "use my data for training." Turning it off usually stops new interactions from being ingested going forward.
OpenAI's own data controls, for instance, let users switch off the setting called "Improve the model for everyone," which stops new conversations being used to train future models, while retained logs and safety-related processing can still occur regardless of that toggle.
Jurisdiction changes the strength of that option considerably. The gap is stark enough to be worth naming directly:
- EU users generally have the strongest position, backed by GDPR data-subject rights. Meta, for example, offers EU users a distinct opt-out from Instagram AI training that isn't mirrored for users elsewhere.
- US users typically lack an equivalent statutory right at the federal level, leaving them dependent on whatever a company voluntarily offers, which varies enormously between providers.
- Other markets sit somewhere in between, often following whichever major regulatory bloc most influences the company's compliance posture.
The opt-out reality check: turning off data sharing is forward-looking only. It stops new data going in; it does nothing to remove influence already trained into a model's weights, and most providers reserve the right to keep processing content for safety, security, or legal compliance even after you've opted out. Malwarebytes' research on chatbot opt-outs found that safety-flagged conversations can still be retained and used to refine safety systems regardless of your training preference. Treat any opt-out as damage control for tomorrow, not a fix for what's already happened.
What privacy-preserving techniques actually work?
Three technical approaches dominate the conversation among engineers trying to reduce training risk without abandoning personalisation altogether.
User-level differential privacy adds carefully calibrated statistical noise so no single person's data can be reverse-engineered from a trained model. Google's research team demonstrated that fine-tuning large language models with user-level differential privacy is workable, but it demands more noise and tighter bounds on how much any one user can influence the outcome than older example-level privacy methods required. The trade-off is real: stronger privacy guarantees usually cost some model accuracy, and tuning that balance is genuinely hard engineering work, not a checkbox.
Federated learning keeps raw data on a user's own device and only shares model updates, not the underlying information, with a central server. Local-first architectures built on this principle can deliver meaningful personalisation without ever centralising raw personal data, which sidesteps a huge chunk of the exposure risk that comes with pooling everyone's data in one place.
Embedding-based personalisation takes a different route entirely: instead of retraining a model on your data, the system builds a compact representation of your preferences and feeds it in at inference time. Google's work on efficient contextualisation with user embeddings shows this can adapt a model's responses to an individual without that individual's raw data ever entering a training pipeline.
- Differential privacy: strong mathematical guarantee, needs careful tuning to avoid hurting accuracy.
- Federated learning: keeps data local, but coordination and update aggregation add engineering overhead.
- Embeddings at inference: fast to deploy, lower privacy exposure, but doesn't improve the base model itself.
Pro Tip: If you're evaluating an AI vendor for sensitive work, ask specifically whether personalisation happens through retraining or through inference-time embeddings. The answer tells you immediately whether your data ever leaves your control.
What risks come from training AI on user data?
Three categories of harm show up repeatedly in security and fairness research, and they don't require malicious intent to occur.
Memorisation and extraction attacks are the most direct threat. Large models can memorise verbatim snippets of training data, and researchers have demonstrated membership inference attacks (proving specific data was in a training set) and model inversion attacks (reconstructing approximations of original inputs) against real systems.
Bias and fairness harms emerge when training data over-represents some groups and under-represents others. A model trained predominantly on one demographic's language patterns or one region's content will perform worse, and sometimes unfairly, for everyone outside that slice.
Confidentiality and downstream licensing risk compounds the problem once a model is fine-tuned or shared. If personal data leaked into a base model, every derivative built on top inherits that exposure, and tracing it back becomes progressively harder the further downstream you go.
- Memorisation: sensitive strings (names, addresses, even credentials) can resurface in model outputs under the right prompting.
- Bias amplification: skewed training data doesn't just reflect existing bias, it can sharpen it in the model's outputs.
- Human review exposure: investigative reporting has documented contractor programmes reviewing user content for quality and safety purposes, sometimes with weaker anonymisation than users assume.
How can individuals reduce their exposure?
Start with the settings you already have access to, then work outward to your habits.
- Audit your account settings in every AI tool you use regularly. Look specifically for a data-sharing or "improve the model" toggle, and check whether it defaults to on.
- Use built-in privacy portals to review, export, or delete stored conversation logs where the option exists, rather than assuming deletion happens automatically.
- Control your website's exposure if you publish content: a
robots.txtfile can block known training crawlers like GPTBot, though compliance is voluntary and not every crawler respects it. Paywalls and takedown requests offer stronger, if more labour-intensive, protection. - Change what you share in the first place. Avoid pasting sensitive personal, medical, or financial details into public chat prompts or social posts; assume anything typed into a free consumer tool could eventually inform a model.
- Move sensitive work to paid or enterprise tiers, which commonly exclude business data from training by contract, unlike many free consumer plans.
Pro Tip: Robots.txt is a request, not a lock. If content is genuinely sensitive, a login wall or paywall stops scraping far more reliably than a crawler directive ever will.
How should organisations govern AI training data?
Individual habits only go so far. Organisations handling other people's data need structural controls, not just settings changes.
Start with a Data Protection Impact Assessment (DPIA) for any project involving AI training or fine-tuning, using tools like Smishalert to help safeguard trust and manage cybersecurity risks. Regulators like Australia's OAIC recommend privacy-by-design practices baked in from the start of a project, not retrofitted after a system is already live. Pair that with a clear data inventory documenting where personal information enters your pipeline and where it flows next.
Vendor contracts deserve particular scrutiny. Push for clauses that explicitly exclude your enterprise data from training future models, grant your organisation audit rights, and specify concrete retention periods and human review rules. Understanding who actually owns data in a SaaS contract matters more than most procurement teams realise until a dispute forces the question.
| Control area | What to implement | Why it matters |
|---|---|---|
| Assessment | DPIA before AI project launch | Surfaces privacy risk before data flows are locked in |
| Contracts | Training-exclusion and audit-rights clauses | Prevents vendor reuse of your data without recourse |
| Operations | Logging, minimisation, retention limits | Reduces what's exposed if a breach or subpoena occurs |
| Access | Secure review workflows, staff training | Limits internal misuse and accidental exposure |
Mapping privacy-by-design principles directly onto engineering backlogs rather than treating them as a compliance memo tends to be the difference between policies that get followed and policies that get ignored.
What should you demand from AI vendors on transparency?
Genuine transparency looks like documentation you can check, not a paragraph of marketing language in a privacy policy.
Ask for model cards and datasheets describing what a model was trained on, at least at a categorical level. Ask whether the vendor has undergone or would permit an independent audit of its training data practices. Neither guarantees perfection, but their absence is a signal worth noting.
- Does the vendor state clearly whether consumer inputs are used for training, and by default or by opt-in?
- What's the actual retention window for logs, and does it differ for free versus paid tiers?
- Are human reviewers involved, and under what anonymisation and access controls?
- Can you request audit logs or contractual proof of data lineage rather than a verbal assurance?
If a vendor can't answer these plainly, treat that vagueness itself as the answer.
How Seven approaches user data in project management
Seven was built on a simple premise: your project data is yours, and it shouldn't quietly become someone else's training material. Seven doesn't mine user data for analytics sales, and it doesn't use customer content to train AI models, a position that shapes the product from the ground up rather than sitting as a privacy-policy footnote.
That shows up in practical features. Flexible workspaces and open data export mean you're never locked into a platform you can't leave with your own information intact. Built-in messaging and file attachments keep team communication inside a system that isn't harvesting conversations for a secondary business model. Excel import makes migration painless rather than a reason to stay put out of inertia.
Pricing stays transparent too: Individual plans run $5 per month, and Teams plans cost $9 per user per month, with no hidden tiers hiding better privacy behind an enterprise sales call. For organisations weighing which tools touch sensitive project data, that's a concrete, checkable difference rather than a promise to take on faith.
What's the honest trade-off here?
Models genuinely do improve with more data. That's not spin, it's how machine learning works, and pretending otherwise doesn't serve anyone. The honest question isn't whether data improves AI, it's whether the governance around that data matches the sensitivity of what's being collected.
What I'd push back on is the idea that opting out solves the problem. It doesn't. It stops one tap flowing forward while everything already collected sits baked into a model's weights indefinitely. Real protection looks less like a toggle and more like a habit: audit your settings quarterly, document where your organisation's data actually goes, and insist on contractual exclusions before you sign, not after you're worried.
— Greg
Sources
For readers who want to verify any claim here directly, start with the OAIC's guidance on generative AI and privacy for regulatory context, and OpenAI's data-controls help page for a working example of vendor settings. Google's research on user-level differential privacy covers the technical mitigation side, while Built In's opt-out guide and Malwarebytes' chatbot opt-out walkthrough offer practical, platform-by-platform steps.
- Guidance on privacy and developing and training generative AI models | OAIC
- How to Opt Out of AI Training: 10 Ways to Protect Your Data | Built In
- How your data is used to improve model performance | OpenAI Help Center
FAQ
Is AI trained on existing data?
Yes. Most large AI models are built on existing public web content, licensed datasets, and user-submitted data during pretraining, then refined further during fine-tuning using narrower, often user-sourced datasets. The exact proportion of user versus public data varies by provider and is rarely disclosed in full.
How do I train AI on my data?
Organisations typically fine-tune an existing foundation model on their own dataset rather than building one from scratch, using techniques like supervised fine-tuning or RLHF to adapt behaviour to specific tasks. For sensitive data, applying user-level differential privacy during that process gives stronger guarantees that no individual record can be reverse-engineered from the result.
How do I keep AI from training on my data?
Turn off data-sharing settings in the tools you use, such as OpenAI's "Improve the model for everyone" toggle, and check for a similar option in any other AI product. For websites, blocking known training crawlers via robots.txt and using paywalls for sensitive content adds another layer, though neither offers a total guarantee.
What is the 30% rule in AI?
There's no single, widely recognised "30% rule" governing AI training data proportions or thresholds; definitions of this term vary depending on the source and context. If you've seen it referenced somewhere specific, it's worth checking that source's own definition directly rather than assuming a standard meaning applies.