LLM Comparator

Intro

Compare responses across multiple large language models with the same prompt set, then review side-by-side output, latency, and token usage.

You will be prompted for your API keys when you run a comparison, or enter them here now.
Conversations can be saved and loaded as JSON or as GitHub Gists
Conversations saved as Gists can be shared as links, for example the improve Python prompt from OpenAI.

Data Security

All prompts, responses, and API keys are stored in your browser's local storage on this device. There is no server-side storage for this tool.

API requests are sent directly from your browser to each LLM provider. There is no intermediary service adding extra security or filtering between you and the provider.

Local Storage Keys

OpenAI → openai_api_key

Anthropic → anthropic_api_key

Gemini → gemini_api_key

OpenRouter → openrouter_api_key

OpenRouter Models → openrouter_models_cache

Custom Providers → custom_providers

Selected Models → selected_models_by_provider

Saved Models → *_saved_models

Prompts → saved_system_prompt / saved_conversation_rounds

GitHub PAT → github_pat

Stored API Keys