10 Terabytes. 1 letter is 2 bytes in unicode. 10e12 bytes. 5e12 letters. 1 word is about 5 letters. 1e12 words. Assuming 500 words per page, that would be 2e9 pages. Page thickness assumed at 1e-4. 2e5 meters, if you stack all the pages. 200 kilometers high tower of paper.
<pedantry>For English language text letters are 1 byte when encoded in UTF-8 (which is how essentially all text is stored and transmitted these days).</pedantry>
Though it probably balances out in your final answer since you’re not counting spaces, punctuation, formatting, or other markup.
10 Terabytes. 1 letter is 2 bytes in unicode. 10e12 bytes. 5e12 letters. 1 word is about 5 letters. 1e12 words. Assuming 500 words per page, that would be 2e9 pages. Page thickness assumed at 1e-4. 2e5 meters, if you stack all the pages. 200 kilometers high tower of paper.
Meaning that to train even the tiniest LLM you can get you need a stack of paper so tall it could get knocked by satellites.
This is the scale that people miss when they talk about “writing some stuff to train an AI model”.
<pedantry>For English language text letters are 1 byte when encoded in UTF-8 (which is how essentially all text is stored and transmitted these days).</pedantry>
Though it probably balances out in your final answer since you’re not counting spaces, punctuation, formatting, or other markup.