Source: Arize Phoenix docs: evals, experiments, phoenix-otel, self-hosting, and Arize pricing
Phoenix's evaluator model (ClassificationEvaluator, Score, pre-built metrics), dataset experiments, register() tracing setup, free ELv2 self-hosting, and Arize AX plan prices.
Contents
- What: official documentation, GitHub READMEs and pricing page
- Publisher: Arize AI
- Date: accessed 2026-11-14
- Original: LLM evals overview, How-to evals, Custom LLM evaluators, Pre-built metrics, Datasets quickstart, phoenix-otel setup, Self-hosting, License, Phoenix GitHub, OpenInference GitHub, Arize pricing
Summary
Phoenix’s eval layer (arize-phoenix-evals) runs code and LLM evaluators client-side or server-side, returning typed Score objects; its experiments run a task over a dataset and score it. Self-hosting is free and unrestricted; Arize AX is the paid SaaS/enterprise product.
Key claims
- Evaluators return
Scorewithname,kind(llm, code, human),direction, and optionalscore,label,explanation,metadata; they exposeevaluate/async_evaluateand aninput_schema. input_mappingmaps evaluator fields to data by key, dot/JSONPath path or callable;bind_evaluatorfixes the mapping.async_evaluate_dataframebatches with an executor that handles rate limits, retries and dynamic concurrency (claimed up to 20x speedup).- LLM judges answer via forced tool calling / structured output, never free-text parsing.
ClassificationEvaluator(name, prompt_template, llm, choices, direction):choicesis a label list or a label-to-score dict.LLM(provider, model)returns label plus explanation;DecisionModelreturns per-label probabilities and no explanation. - Each evaluator run is itself traced to a dedicated Phoenix project (judge prompt, reasoning, score, timing).
- Pre-built LLM metrics: faithfulness, hallucination, completeness, conciseness, correctness, retrieval relevance, tool selection, tool invocation, tool response handling, refusal, user friction, PII, toxicity; code metrics: exact match, regex, precision/recall/F1. Arize states its templates reach F1 of 85%+ on its golden datasets (vendor claim).
- Experiments:
create_dataset(name, examples)thenrun_experiment(dataset, task, evaluators, experiment_name); results compared in the dataset’s Experiments tab. TypeScript equivalents in@arizeai/phoenix-client. phoenix.otel.register(project_name, auto_instrument, batch, endpoint, protocol, headers); Python defaults tobatch=False, TypeScript tobatch: true(callshutdown()to flush). Env vars:PHOENIX_COLLECTOR_ENDPOINT,PHOENIX_API_KEY,PHOENIX_PROJECT_NAME; local server atlocalhost:6006.- Packages:
arize-phoenix(server),arize-phoenix-otel,arize-phoenix-client,arize-phoenix-evals; TS@arizeai/phoenix-otel,-client,-evals(alpha),-cli(px setuphands instrumentation to a coding agent). - OpenInference (Apache-2.0) instrumentors exist for Python, JS, Java and Go and export to any OTel collector; span processors translate OpenLIT and OpenLLMetry traces.
- Self-hosting: free, no usage limits or feature gates, air-gappable; OAuth2/LDAP/local auth, RBAC, retention policies; Docker (
arizephoenix/phoenix), Helm, CloudFormation; SQLite by default, PostgreSQL for production. Licence is Elastic License 2.0 (the docs call it open source; ELv2 is source-available, not OSI-approved). - Arize AX pricing (2026-11-14): Free (25k spans/month, 1 GB, 15-day retention), Pro $50/month (50k spans, 10 GB, 30 days), Enterprise custom with SaaS or self-hosted deployment.