Guide
How AI companies get training data
How do AI companies get training data? From five sources: web crawls, licensed archives, paid human work, synthetic data and their own users.
Published
The short answer
AI companies get training data from five main sources: public web crawls such as Common Crawl, licensing deals with publishers and platforms, paid human work like labelling and expert tasks, synthetic data generated by other models, and their own users' content under their terms of service. As public text runs short, licensed and private data matter more.
The five sources at a glance
| Source | Public example | Main risk | Where a broker fits |
|---|---|---|---|
| Web crawls | Common Crawl, collected since 2008 and free to access | Copyright claims, low quality, duplicates | Little: the data is already public |
| Licensing deals | Reddit disclosed licensing agreements worth $203.0 million in aggregate in its 2024 IPO filing | Rights chains, exclusivity terms | Large: finding owners and structuring licences |
| Paid human work | Meta's reported $14.3 billion investment for 49% of Scale AI in June 2025 | Quality control, worker conditions | Some: sourcing expert task sets |
| Synthetic data | Microsoft researchers trained phi-1 partly on synthetic textbooks written by GPT-3.5 | Errors copied from the generating model | Little |
| Their own users | Meta's April 2025 plan to train on EU adults' public posts, with an objection form | Privacy law and user backlash | None |
- Common Crawl has collected the web since
- 2008
- Reddit's disclosed licensing agreements, in aggregate (2024 IPO filing)
- $203.0 million
- of Scale AI, for Meta's reported $14.3 billion (June 2025)
- 49%
1. Crawling the public web
The largest early language models were trained mostly on text crawled from public web pages. Common Crawl, a non-profit, has collected web pages regularly since 2008 and offers petabytes of raw pages, metadata and extracted text, free to access on Amazon's cloud.
Labs filter that raw material heavily, removing duplicates, spam and low-quality pages, and many run their own crawlers too.
The limits of crawling
The limits are well known. Much of the web is repetitive or poor, many sites now block AI crawlers, and the copyright status of training on scraped work is being argued in courts around the world. Public data is also available to every competitor, so it can't make one model better than another on its own.
2. Licensing archives from their owners
Licensing is the clearest sign that data has become a traded asset. Reddit's IPO filing disclosed that in January 2024 it entered data licensing arrangements with an aggregate contract value of $203.0 million and terms of two to three years.
Reuters reported, citing unnamed sources, that one of them was with Google at about $60 million annually; the companies confirmed a partnership but not the price.
Beyond social platforms
Deals like these go beyond social platforms: news publishers, stock media libraries, book publishers and specialist data owners have all licensed archives. What buyers pay for is content they can't get from a crawl, with clear rights and good documentation. Private data such as codebases, support conversations or domain records fits that description, which is why it interests labs.
3. Paying people to make or label data
Raw text teaches a model language; human work teaches it to follow instructions, reason and avoid mistakes.
Labs pay contractors and specialist firms to label images, rank answers, write example conversations and solve expert tasks in coding, law, medicine or finance.
The scale of the market
The scale of that market showed in June 2025, when Meta took a 49% stake in data-labelling company Scale AI in a deal reported at $14.3 billion.
Human data is expensive and slow, and its value depends on the expertise of the people producing it. That is one reason labs also look for task data that already exists: real work products, with a prompt and a solution, created inside companies over years.
4. Generating synthetic data
Models can now write training material for other models. In the phi-1 paper, Microsoft researchers trained a small coding model on about 6 billion tokens of filtered web data plus about 1 billion tokens of synthetic textbooks and exercises generated with GPT-3.5.
Synthetic data is cheap and can target gaps, but it inherits the mistakes and blind spots of the model that wrote it, so labs mix it with real data rather than replace it.
5. Using their own users' content
Companies with large platforms can use what their users create, within the limits of their terms and privacy law. In April 2025 Meta announced it would train its models on public posts and comments from adults in the EU, notify users and offer a form to object, while excluding private messages and accounts of under-18s.
Whether this is lawful depends on the jurisdiction and on what users were told, and it is the source most likely to cause backlash.
How to use this if you hold or broker data
- 1
Place the data in one of the five sources
If it is already public, a lab can probably crawl it, and its value is low. Private, licensed or expert-made data is where buyers pay.
- 2
Check the rights chain
Confirm the owner created the data or holds rights to license it, and that customer terms, contributor agreements or open-source licences don't forbid the use.
- 3
Remove or protect personal data
Strip personal details where possible and record what was removed. Labs will ask, and privacy law may require it.
- 4
Document it like a product
Size, format, time span, how it was collected, known gaps, and a sample. Good documentation shortens a buyer's review.
- 5
License, don't sell outright
Most data deals are licences with limits on use, term and exclusivity. Agree these before price, and have a lawyer review the agreement.
Limits of this guide
Labs rarely publish full details of their training data, so this guide relies on public filings, research papers and reporting rather than inside knowledge.
The deal values quoted are the ones disclosed or reported; most deals are never priced publicly, and the figures don't tell you what any other dataset is worth.
Law on training with copyrighted and personal data is changing fast, so check the current position before acting on it.
Sources
6 sources · checked Nov 7, 2026
The DataBroker team checked the facts on this page against 6 public sources, last on .
- Common Crawl overviewCollected since 2008, petabytes of data, free access on Amazon · Checked
- Reddit, Inc. Form S-1 (SEC, February 2024)$203.0 million aggregate licensing contract value, two to three year terms · Checked
- TechCrunch on Reddit's S-1 and the Reuters reportReported Google deal at about $60 million annually, unconfirmed price · Checked
- Bloomberg: Meta finalizes Scale AI investment$14.3 billion investment, 49% stake, June 2025 · Checked
- Textbooks Are All You Need (arXiv 2306.11644)phi-1 training mix of web and synthetic textbook data · Checked
- Meta: Making AI work harder for Europeans (April 2025)Training on EU adults' public content, objection form, exclusions · Checked
How AI companies get training data: FAQ
Do AI companies pay for training data?
Often, yes. Alongside crawling public pages, labs license archives from publishers and platforms and pay firms and contractors for labelling and expert tasks. Some deals are public, such as the licensing arrangements Reddit disclosed in its IPO filing, but most prices are never published.
Is it legal for AI companies to train on scraped web data?
It is contested. Courts in several countries are hearing copyright cases about training on scraped work, and privacy law limits the use of personal data. The answer depends on the country, the data and how it was used, and it is still changing.
Are AI companies running out of training data?
Researchers have warned that high-quality public text is limited compared with how fast models consume it. That is one reason labs turn to licensed private data, paid human work and synthetic data.
Can my company sell its data to AI companies?
Possibly, if the data is private, useful to a model, and yours to license, with personal details removed or properly consented. Codebases, task data, domain records and media archives are common examples. Start by checking the rights before you talk about price.
What is synthetic training data?
Data written or generated by an AI model to train another model, such as example exercises or conversations. It is cheap and targeted but can repeat the generating model's errors, so it is usually mixed with real data.
Related
- How to sell data to AI companies
Sell data to AI companies only after you can prove where it came from, that you may license it, and why it isn't online already.
How to become a data brokerHow to become a data broker who brings company datasets to buyers: the skills, the legal checks, the documents and the order you do them in.
- Data valuation
Data valuation explained for people selling or brokering a dataset: three standard methods, what each misses, and a worksheet to score one.
- Is selling data legal?
Is selling data legal? A plain look at when a sale is allowed, when registration applies, and when to call a lawyer first.
- Data broker
The data broker definition in plain words and in state law, the six kinds of data broker, and the rules each kind answers to.
Join the DataBroker academy
The DataBroker academy and tools: find companies with data to sell, price it, license it and close the deal.








