Skip to content
Sean Bettsv2.0.0
Back to Building
research toolCompleted

LLM Search Analysis

A research tool for comparing how AI models search the web, which sources they find and what they cite in their answers.

Sole developer · November 2025View on GitHub (opens in a new tab)
Recorded GPT-5.1 interaction from 1 December 2025: a question about the latest Steam Controller information produced the query Valve Steam Controller discontinued, with 20 search results and 9 cited sources. One saved case, not a current model benchmark.
In practice

What it does

  1. Compare responses to the same question across OpenAI, Google Gemini and Anthropic Claude.
  2. Inspect the search queries, returned sources and citations each provider exposes.
  3. Investigate ChatGPT through browser captures to examine details beyond those available through the APIs.
  4. Run batches of questions, revisit previous results and export the data for further analysis.
Behind the work

The challenge

I wanted to understand what happens between asking an AI assistant a question and receiving an answer with sources. I built the interface, backend and data collection tools myself, combining provider APIs with browser captures of ChatGPT. Each exposes a different view of the search process. The difficult part was making useful comparisons while keeping gaps in the evidence visible.

What I learned

This project changed how I think about visibility in AI search. In the examples I investigated, the sources selected for the answer appeared to reflect the meaning behind the original prompt more closely than a literal match to the search query. For brands, that shifts the question from which keywords their content targets to how well it answers what someone is actually trying to find out.

Built with

  • Python
  • Streamlit
  • FastAPI
  • Playwright
  • SQLite
  • SQLAlchemy
  • OpenAI API
  • Google AI API
  • Anthropic API
  • pandas