Streamsight

What I built

Streamsight is a retrieval-augmented generation (RAG) system I built as a proof of concept inside a large media company. The goal was to take a large body of internal workplace insights — survey results, strategic documents, project retrospectives — and make them genuinely searchable and queryable, rather than sitting inert in document repositories that nobody opens.

It uses Google Gemini for the language model layer, a custom indexing pipeline to process and embed the source documents, and a basic authentication layer to restrict access to appropriate users. The front end is intentionally minimal — the point of the proof of concept was to validate the retrieval quality, not to build a polished product.

What problem it solved

Large organisations accumulate a lot of institutional knowledge that becomes effectively inaccessible once it’s written down. The insight from six months ago that would directly inform the decision being made today is sitting in a SharePoint folder nobody remembers exists. Streamsight was an attempt to make that body of knowledge queryable in natural language.

The specific use case was employee listening data — the kind of structured insight that organisations collect and then struggle to act on because synthesising hundreds of responses is time-consuming and the nuance tends to get lost in the summarisation.

What I learned

The hardest part of RAG isn’t the retrieval or the generation — it’s the indexing. Getting the chunk size and overlap right, handling documents with different structures, ensuring that the metadata attached to each chunk is accurate enough to support useful filtering — these are unglamorous problems that have a huge impact on output quality.

The second big lesson: users need to trust the outputs before they’ll change their behaviour. Showing your working — surfacing the source documents that informed a response — matters more than the quality of the generated text itself. Transparency is a feature.

Tech used

Google Gemini handles the generation layer. The indexing and retrieval pipeline is Python, using standard embedding libraries with a vector store for similarity search. Authentication is handled separately and keeps the POC out of reach of general staff while the approach is being validated.