The FileAnalyzer is a utility for large scale file ingestion. It helps with generating context for identified, white-listed mime-types, typically as a part of the lifecycle generating large data sets for LLM’s.

It adds meta data, splitting the content in chunks and stores the information in an intermediate vector database, while using efficient caching and transactional mechanisms to ensure safe start/stop/start sequences.

Documentation for the FileAnalyzer tool.