Over the past few years, we've seen an increasing number of clients approaching us with the same fundamental challenge: they're drowning in unstructured documents and struggling to extract meaningful insights from them. Whether it's insurance companies with thousands of claim files, law firms with decades of case documents, or healthcare organisations with extensive patient records, the problem is consistent - valuable knowledge is trapped in PDFs that are impossible to navigate efficiently.
Traditionally, our approach involved keyword-based indexing systems, often built with tools like Elasticsearch. We'd extract text, build searchable indexes around key terms, and implement filtering based on predefined categories. While this worked for basic retrieval, it had significant limitations: documents could only be found if users knew exact terms to search for, manual categorisation risked misclassifying documents, and keyword matching couldn't understand meaning or context. A claim involving "vehicle damage" might be missed if someone searched for "car accident," and documents with similar intent but different terminology would never be connected.
The emergence of large language models and vector databases has fundamentally changed this landscape. By combining Claude's language models for classification and analysis with OpenAI's embeddings API for vector search using sqlite-vec, we can now build systems that understand meaning rather than just matching words. Documents can be classified automatically based on semantic content, and similarity search can discover connections that would never emerge through traditional keyword approaches - turning static repositories into intelligent, discoverable knowledge bases.
The first challenge is converting unstructured PDFs into machine-readable text while preserving semantic meaning. Our pipeline uses the pdf-text-extract library to handle various PDF document formats and extract readable text content.
The extracted PDF text is processed by combining multiple pages into a single continuous text block. This approach preserves the document's narrative flow while creating a format suitable for AI analysis. The raw text is stored directly in the SQLite database alongside other case study metadata for subsequent AI processing.
We use Claude for document analysis because it has a longer context window, so can handle much larger documents without needing to chunk them down. While it is the right tool for the first step of classification, it doesn’t have a dedicated embedding API, so we used OpenAI for the second part of the classification.
The system uses Claude to analyse document content and extract structured information, including:
Each processed document is converted into high-dimensional vectors using OpenAI's text-embedding-3-small model. The embedding input combines Claude's structured outputs (summary, group, service, key points, and outcome) into a single text string that gets converted into 1536-dimensional vectors capturing semantic meaning.
The system uses the generated embeddings to find similar cases through:
This approach leverages Claude's language understanding to create consistent, structured data that OpenAI then converts into searchable vector representations for scalable similarity matching.
We chose sqlite-vec for its simplicity and performance characteristics:
The vector database enables sophisticated queries:
To prove out this theory, we decided to stand up a proof-of-concept application using modern, accessible technologies that could demonstrate the practical viability of semantic document classification and similarity search.
The proof-of-concept follows a straightforward data pipeline: documents are uploaded and processed through PDF text extraction, content is cleaned and sent to OpenAI for classification and embedding generation, then stored in SQLite with vector extensions for fast similarity search. This architecture prioritises simplicity and deployability while maintaining the core functionality needed to validate the semantic search approach.
By choosing lightweight, familiar technologies like SQLite and standard web frameworks, we created a system that can be deployed anywhere without complex infrastructure requirements, making it easy to demonstrate and iterate on the core concept with real document collections.
We've brainstormed some use cases where we thought this approach would be particularly useful, especially for organisations dealing with large volumes of similar documents that need to be categorised and matched against historical patterns.
Each application uses AI classification and vector similarity search to transform manual document processing into intelligent, automated workflows.
Instead of spending hours searching through filing cabinets or digital folders, hoping to find the right precedent, staff can now find similar cases in seconds. An insurance adjuster dealing with a complex motor vehicle claim immediately sees how 20 similar cases were resolved, their settlement amounts, and processing times - turning guesswork into data-driven decisions.
Different team members no longer categorise the same type of case differently based on their personal judgment. The AI applies the same logical analysis every time, meaning a new hire and a 20-year veteran will get the same classification results. This consistency improves service quality and reduces customer complaints about inconsistent treatment.
The system reveals connections that humans would never spot. For example, it might discover that certain types of health insurance claims submitted during winter months have different approval patterns, or that customer complaints mentioning specific phrases tend to escalate more often. These insights help businesses proactively address issues before they become problems.
Manual document review and categorisation that previously took days now happens automatically. Staff time is freed up for higher-value work like customer service, complex case analysis, or business development.
Traditional systems like Elasticsearch require users to guess the exact words used in documents. Our approach understands meaning, searching for "car accident" will find documents about "vehicle collisions," "motor incidents," or "automotive damage", even if those exact words weren't used. This eliminates the frustration of knowing relevant information exists but being unable to find it. ####More Portable Than Proprietary Index-Only Approaches A common pain point with traditional search engines is the effort required to build and tune indexes that are tightly coupled to a specific platform. To reduce that lock-in, we separate AI classification from vectorisation and persist classification outputs (such as categories, entities, summaries, and key points) as reusable metadata, for example in JSON sidecar files alongside the source documents. If we later move to a different vector store, we can usually re-vectorise from this enriched data without re-running full document classification.
Unlike traditional filing systems that require constant manual updates to keyword lists and categories, this AI-powered approach automatically adapts to new document types and terminology. When new regulations create new claim types or industry language evolves, the system learns and adjusts without requiring technical maintenance.
Processing 1,000 documents takes roughly the same effort as processing 100,000, so the system doesn't slow down or require proportionally more staff as volume grows. Traditional manual classification would require hiring more people; this approach handles growth automatically.
This semantic understanding fundamentally solves the biggest problem with traditional document management: valuable information getting lost simply because someone didn't use the "right" search terms or file it in the expected category.
This project demonstrates that AI-powered document classification and similarity search are practical solutions using accessible technologies today. By combining Claude’s language models with OpenAI's embeddings API and sqlite-vec, organisations can transform manual document categorisation into intelligent, automated systems that reveal hidden patterns and accelerate decision-making. This approach is particularly valuable for insurance claims processing, collections management, medical diagnosis support, and customer service optimisation, anywhere finding the right precedent or similar case significantly impacts outcomes.
If your organisation is struggling with unstructured data trapped in documents and traditional search methods aren't delivering the insights you need, Abletech can help. We specialise in building practical AI solutions that turn your document collections from static repositories into intelligent, searchable knowledge bases that drive better decision-making across your organisation.
read previous