Extractium™

From scattered content to shared knowledge.

Gather your websites, knowledge base, code, and video captions into one compendium. Search it from a web page, a script, or any AI assistant, with no server to run.

Build your first compendium Explore the code
  • Portable files
  • Keyword and meaning-based search
  • Your choice of AI provider
Documents from websites, repositories, libraries, and videos come together in an Extractium book, then support people using search, AI assistants, and reports.
Websites, repositories, libraries, and videos, gathered once and reused in search, AI assistants, and reports.

Index what you already publish

  • Websites & portals
  • GitHub repositories
  • Library repositories
  • YouTube captions
  • Local folders
  • Knowledge bundles
What you get

Website search, llms.txt, SQL,
and Markdown, from one build.

Extractium gathers and prepares your content once, then writes it in several formats. The output is ordinary files you can keep, share, or host, including on GitHub Pages.

Add search to your website

compendium.json.gz
compendium-full.json.gz

Keyword and meaning-based search from one file, in a page, a script, or an AI assistant on your computer.

How to search a compendium

Publish llms.txt for AI assistants

llms.txt
llms/

Give web-browsing assistants your content in the form they read, and keep a readable list of what was indexed.

How to use the published files

Query it with SQL

compendium.sqlite

Build reports from the database file, or load it into a hosted database for a remote assistant.

How to host search remotely

Read and edit it as Markdown

okf/

A folder of Markdown files in the Open Knowledge Format, for reading, editing, and other tools.

Choose which outputs to write

No database service or always-running server is required to build the files. Source and output plug-ins let you extend the formats.

Try it with one website

Your first build,
one step at a time.

Start small. The setup script asks three questions and builds an index of up to 25 pages for you to review.

  1. Install Python

    You need Python 3.10 or newer. The script sets up Extractium’s dependencies in its own environment.

  2. Make a folder and save the script

    Create an empty folder for your compendium. Use the script link for your operating system, then save that file in the folder.

  3. Run it and answer three questions

    Open a terminal in that folder and run the command shown. Give your compendium a name, a short file name, and a starting website.

  4. Review, then build the rest

    Open dist/llms.txt, then the file it links to under dist/llms/, to review the pages. Run the same command again to build the full site. Edit config.yaml when you want to adjust the sources.

Allow about 1 GB for dependencies, plus a 130 MB model downloaded on the first build. Later builds reuse the model. Read the installation guide for requirements and other ways to install.

Start on your computer

Windows · PowerShell or Command Prompt

Open the Windows script

Save the linked file as run.bat in your new folder, not as a .txt file.

Run this command in that folder:

.\run.bat

macOS / Linux · Terminal

Open the macOS / Linux script

Save the linked file as run.sh in your new folder, not as a .txt file.

Run this command in that folder:

bash ./run.sh

Then answer the setup prompts:

  1. A name for your compendium.
  2. A short name for its files.
  3. The website to start with.
The script downloads Extractium, sets it up, and runs the first build.

Your build output stays in the folder you chose. Publishing them is a separate step. Local-folder content is excluded from each output unless you explicitly enable it.

See the Depression Center’s own settings file
Build once. Keep your options.

Yours to host, move,
and build on.

The same files serve a website search box, your own analysis, and an AI assistant. You decide where they live.

Browse the documentation

  • Publish search beside your pages

    Host the index next to your website and let the JavaScript client answer searches in the browser.

  • Connect an AI assistant

    Point a web-browsing assistant at llms.txt, or connect one to the local or hosted search server.

  • Build on it

    Read the files with the Python and JavaScript clients, or add a source or output plug-in of your own.

Extractium is developed by the Mobile Technologies Core at the Eisenberg Family Depression Center, University of Michigan, and grew out of the indexing engine in Field Station AI™.

Licence

The project is open source under the GNU General Public License v3.0 or later. Its documentation uses the GNU Free Documentation License v1.3 or later.

Three questions, one command,
twenty-five pages.

The first build is small on purpose. See what it includes, then build the rest.

Run the quick start