How FirmRanker measures AI visibility

Study, experiment, observation

Study → Experiment → Observation → Validation → Analysis

These distinctions matter because claims belong at different levels. An observation shows what happened once. An experiment groups observations under the same conditions. Only a study, with its stated scope, supports a finding — and only within that scope.

• Study — A defined research programme with a specific question, population and methodology.
• Experiment — A particular combination of prompt, market, practice area and AI platform/model.
• Observation — The saved AI response from a specific run.

How prompts are designed

A prompt is part of the experimental condition. FirmRanker designs prompts to reflect the ways people actually ask for legal help, and records each one exactly. Prompts are not rewritten after collection.

Unbranded prompts ask which firms to consider for a legal need without naming any firm. Branded prompts ask about a specific firm. They answer different questions and are analysed separately.

• Direct recommendation — “Which family law firm in Sydney would you recommend?”
• Comparison — “How do the leading employment firms in London compare?”
• Problem-based — “I was injured at a shopping centre — who should I talk to?”
• Reputation — “Which personal injury firms in New York have the best reputation?”
• Shortlist / high intent — “Give me three firms to call this week about a custody dispute.”
• Geography and client context — Location, circumstances and urgency are varied deliberately, not incidentally.

Prompt intent and response treatment

Prompt intent ≠ response treatment.

Prompt intent is the kind of response the user's question appears to request — a direct recommendation, a comparison, a list of reputable firms or a problem-based shortlist.

Response treatment is what the AI actually does with each firm. A recommendation-seeking prompt can produce a neutral list; an informational prompt can produce an endorsement. The two are stored separately so that one is never used as a proxy for the other.

Response treatment taxonomy

Mentioned → Suggested → Recommended

• Mentioned — The firm appears substantively but is not placed into an actionable consideration set.
• Suggested — The firm is presented as one the user may consider.
• Recommended — The system explicitly recommends or endorses the firm.
• Source-only — The firm or its domain appears in source material but is not substantively surfaced.
• Negative / cautionary — The firm appears in a materially adverse or cautionary context.

Why mentioned ≠ recommended

A law firm can appear in an AI answer without being recommended.

Prompt

Which personal injury firms in Sydney would you recommend?

Response

Several firms operating in Sydney include Firm A, Firm B and Firm C.

Contrast

I would recommend starting with Firm A and Firm B.

The first response lists firms but may not constitute an explicit recommendation. The second does. Counting both as “recommended” would overstate how often AI systems endorse firms and hide real differences between systems that hedge and systems that commit.

Position

FirmRanker records mention position and, separately, recommendation position. Position can be meaningful in an ordered list explicitly presented as a priority order. It is often not meaningful in grouped answers, tables, alphabetical lists, or lists where the system states no preference.

Order in an answer is recorded as evidence. It is not automatically converted into a ranking.

Entity resolution

The same law firm can appear under many names: its full legal name, an abbreviation, a trading name, punctuation differences, “&” versus “and”, an old name, or an AI-generated variant. FirmRanker preserves the original text while resolving variants to a canonical entity for analysis.

• Confirmed — Resolved with sufficient certainty.
• Probable — Likely match pending review.
• Unresolved — Not yet matched; may not be a real firm.
• Split / merge — Corrections that separate or combine entities, logged with reasons.
• Human approval — Ambiguous decisions are made by reviewers, not guessed.

Law firm, lawyer or source?

Not every capitalised legal name is a law firm. Extraction distinguishes law firms from individual lawyers, legal directories, publishers, regulators, government bodies and entities that appear only in sources. Each type is recorded differently so that, for example, a bar association or a directory is not counted as a competing firm.

Sources

Retrieved ≠ cited ≠ associated ≠ causal.

Some Gemini source URLs pass through Google/Vertex redirect infrastructure. FirmRanker may preserve the raw provider URL, the resolved publisher URL and the resolution status. If resolution fails, it is flagged — not guessed.

• Search result / retrieved source — Returned during provider search or retrieval.
• Cited source — Explicitly cited or linked in the answer.
• Resolved destination — The actual publisher page after redirect resolution.
• Associated source — A source linked to a particular law firm.

Raw answer preservation

Raw answers remain immutable. Corrections apply to extraction, entity resolution, classification and analysis — never to the original observation.

• Auditability
• Reproducibility
• Correction history
• Dispute resolution
• Future re-analysis

Repeatability

One answer isn't a ranking.

AI outputs vary. Repeated observations may be required to understand appearance frequency, recommendation frequency, stability, prompt coverage, model coverage and cross-model consensus. There is no single universal number of repetitions; the study design determines it, and each study states its own.

Model versions

Providers change models. FirmRanker preserves the returned model identity where available, and research states the provider, model, date and retrieval/web status. “ChatGPT” is not treated as one fixed system over time.

Validation

Automated extraction produces an original machine result, which is frozen. Assisted review may propose corrections; final human review creates a gold-standard comparison layer. Machine accuracy is then measured against the frozen original output.

Metrics may include extraction precision, recall and F1, recommendation precision and recall, classification accuracy and confusion matrices, position accuracy, entity-resolution accuracy and source accuracy where measurable. No accuracy metric is published before human QA is complete.

Limitations

These limits shape what any AI visibility measurement can claim. Stating them is part of the method.

• Generative variability — The same prompt can produce different answers.
• Model changes — Providers update models without notice.
• Retrieval changes — Search indexes and retrieval behaviour change.
• Location and personalisation — Context can change results.
• Prompt sensitivity — Small wording changes can change the consideration set.
• Time and the web — Observations are point-in-time; source content changes.
• Provider transparency — Providers expose sources and searches inconsistently.
• No access to reasoning — We observe outputs, not the model's internal process.
• No causal source weighting — Citation cannot reveal how much a source influenced an answer.
• Population and sample limits — Findings apply to the tested markets, practices and prompts.
• Formatting differences — Providers present answers differently, which affects extraction.

Versioning

This is methodology v1.0, last updated 6 October 2026. Future changes will be versioned, and each study will state the methodology and extraction version it used.