Etymonline provides accounts of the origins and histories of English words, phrases and idioms.
General Index
The General Index provides three tables derived from more than 107 million journal articles, including an n-gram table.
About General Index
General Index is a dataset published on the Internet Archive by Public Resource. It consists of three tables derived from 107,233,728 journal articles: an n-gram table ranging from unigrams to 5-grams extracted with SpaCy, a keyword table built with Yake, and a table associating article IDs with metadata.
The n-gram table has 355,279,820,087 rows, each pairing an n-gram with a journal article ID; the keyword table has 19,740,906,314 rows, each pairing a keyword with an article ID. The corpus is split into 16 slices plus an update with additional language files, and keyword and n-gram files are distributed as separate Internet Archive items to avoid overloading servers. Files are large and inflate greatly upon unzipping. The underlying articles are not released. Metadata, readme, and sample files are available in the item's data downloads area.
Key features
- Three tables derived from 107,233,728 journal articles
- N-gram table from unigrams to 5-grams, extracted with SpaCy
- N-gram rows pair each n-gram with a journal article ID
- Keyword table built with Yake, pairing keywords with article IDs
- Third table associates article IDs with metadata
- Corpus split into 16 slices plus additional-language update
Best for: Researchers and developers conducting text and data mining on scientific literature who need large n-gram or keyword tables rather than full article text.
Good for
- Covers 107,233,728 journal articles, with 355,279,820,087 n-gram rows and 19,740,906,314 keyword rows.
- Released under CC0 1.0 Universal with no rights reserved.
- Described as non-consumptive: underlying journal articles are not released.
Watch out for
- Files are large and inflate greatly upon unzipping.
- Data is distributed across 16 slices, an update, and separate keyword and n-gram items.
General Index at a glance
- Category
- Research & Data › Research & Study Tools
- Type
- Website
- Pricing
- Not confirmed. Check the site
- Platforms
- Android, iOS, Web
- Open source
- Not stated
- Language
- English
- Last checked
- Oct 6, 2026 · Online when checked
- Listed
- Oct 5, 2026
General Index alternatives
Tools and sites you could use instead, ranked by how closely they match.
Infoplease combines an encyclopedia, almanac, atlas, biographies, dictionary and thesaurus for online reference.
Open Syllabus maps college curricula across millions of syllabi and provides course matching and analytics.
RefSeek searches academic sources including web pages, books, encyclopedias and journals.
Publish or Perish retrieves and analyzes academic citations, presenting metrics such as publication counts, citations and h-index.
Refdesk organizes quick reference links to facts, encyclopedias, newspapers and subject indexes.
Popular in Research & Study Tools
Well-known listings from the same topic.
NoteKnightnoteknight.comNotebookCastnotebookcast.comGeoGebrageogebra.orgWikitiawikitia.comPapertagpapertag.appMoby Thesaurusmoby-thesaurus.org
General Index FAQ
What does the General Index contain?
It contains three tables derived from 107,233,728 journal articles: an n-gram table from unigrams to 5-grams, a keyword table built with Yake, and a table associating article IDs with metadata.
Are the underlying journal articles included?
No. The General Index is described as non-consumptive, and the underlying articles are not released.
How are the data files organized?
The corpus is split into 16 slices plus an update with additional language files. Keyword and n-gram files are kept as separate Internet Archive items.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.