Skip to main content

Running inference

You run inference through chat.completions.create, which has the same shape as the OpenAI chat completions API. Pass model="auto", a list of messages, and you get a ChatCompletion back. Set stream=True and you get an iterator of token deltas instead.

Pareta is OpenAI-compatible on the wire, so you can run inference with this SDK, with the openai package, or with raw HTTP, whichever fits your stack. This SDK's extra value is the control plane (evals on your own data, auto metrics); for plain inference the two are interchangeable.

A few platform truths that shape this page:

  • There is no model to pick. model is the literal string "auto"; Pareta routes each request behind it. Real open-weights model ids never reach you; the backend resolves them. You never pick a GPU.
  • Inference is metered against your org balance. A successful completion debits your balance — one debit per request, no matter how many internal model calls auto's plan makes. If the balance is empty, the call raises InsufficientCreditsError (402). Top-up is browser-only; the SDK has no balance or payment surface.

model="auto" — the routing brain​

The model id for every request is the literal string "auto". Pareta decomposes the request, routes each part to the cheapest model that holds frontier-grade quality, verifies checkable outputs (escalating to a frontier model on a failed check), and synthesizes one answer. One request, one debit; a request that errors out bills $0. Streaming works the same way — the answer streams token by token, and the SSE stream carries : pareta-progress <stage> comments (planning / executing / answering) you can surface as status.

completion = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "…"}],
)

Everything below — setup, streaming, async, errors — is that one call in different shapes.

Setup​

Mint a pareta_sk_ key in the dashboard, export it, and build the client from the environment:

export PARETA_API_KEY=pareta_sk_...

Python

from pareta import Pareta

pa = Pareta.from_env()   # reads PARETA_API_KEY (+ optional PARETA_BASE_URL)

TypeScript

import { Pareta } from "pareta";

const pa = Pareta.fromEnv();   // reads PARETA_API_KEY (+ optional PARETA_BASE_URL)

from_env() is the recommended path. You can also pass the key explicitly: Pareta(api_key="pareta_sk_..."). The client is a context manager, so with Pareta.from_env() as pa: cleans up the HTTP connection for you.

A basic completion​

Pass model="auto" and a non-empty messages list in OpenAI format. You get back a ChatCompletion.

Python

from pareta import Pareta

with Pareta.from_env() as pa:
    resp = pa.chat.completions.create(
        model="auto",   # the routing brain — the only model id
        messages=[
            {"role": "system", "content": "You extract structured fields from documents."},
            {"role": "user", "content": "What is the invoice total?\n\nINVOICE\nTotal due: $4,210.00"},
        ],
    )

    print(resp.choices[0].message.content)
    print(resp.usage.total_tokens, "tokens")

TypeScript

import { Pareta } from "pareta";

const pa = Pareta.fromEnv();
const resp = await pa.chat.completions.create({
  model: "auto",   // the routing brain — the only model id
  messages: [
    { role: "system", content: "You extract structured fields from documents." },
    { role: "user", content: "What is the invoice total?\n\nINVOICE\nTotal due: $4,210.00" },
  ],
});

console.log(resp.choices[0].message.content);
console.log(resp.usage.totalTokens, "tokens");

model and messages are both required. The SDK raises ValueError before sending if model is falsy or messages is empty, so a malformed call fails fast without burning a request.

The ChatCompletion shape​

create() returns a ChatCompletion. The fields mirror OpenAI:

Python

resp.id                              # str | None
resp.model                           # str | None: echoes "auto"
resp.created                         # int | None: Unix timestamp
resp.choices                         # list[Choice]
resp.choices[0].index                # int | None
resp.choices[0].finish_reason        # "stop", "length", ...
resp.choices[0].message.role         # "assistant"
resp.choices[0].message.content      # str | None: the generated text
resp.usage.prompt_tokens             # int | None
resp.usage.completion_tokens         # int | None
resp.usage.total_tokens              # int | None

TypeScript

resp.id                              // string | null
resp.model                           // string | null: echoes "auto"
resp.created                         // number | null: Unix timestamp
resp.choices                         // Choice[]
resp.choices[0].index                // number | null
resp.choices[0].finishReason         // "stop", "length", ...
resp.choices[0].message.role         // "assistant"
resp.choices[0].message.content      // string | null: the generated text
resp.usage.promptTokens              // number | null
resp.usage.completionTokens          // number | null
resp.usage.totalTokens               // number | null

Every response object keeps the raw server JSON. If a field isn't surfaced as a typed property, reach it with resp.to_dict() or resp["..."]. Nothing the API returns is lost behind the typed layer.

Passthrough parameters​

Any extra keyword you pass goes straight into the request body, so the full OpenAI parameter set is available without the SDK enumerating it:

Python

resp = pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarize this contract clause: ..."}],
    temperature=0.2,
    max_tokens=512,
    top_p=0.9,
)

TypeScript

const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Summarize this contract clause: ..." }],
  temperature: 0.2,
  max_tokens: 512,
  top_p: 0.9,
});

temperature, max_tokens, top_p, stop, seed, and friends all pass through unchanged.

Structured outputs (response_format)​

model="auto" supports OpenAI Structured Outputs: pass a response_format and the response content is guaranteed to be a single JSON document that conforms — however the request was served. Pareta enforces the schema with constrained decoding on its own models, validates every answer against your schema before delivery, and escalates automatically when an answer doesn't conform. If no conformant answer can be produced at all, the request fails with an error and you are not billed — nonconformant JSON is never delivered.

Python

resp = pa.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Classify: 'refund my order'"}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "intent",
            "schema": {
                "type": "object",
                "properties": {
                    "label": {"type": "string", "enum": ["refund", "billing", "other"]},
                    "confidence": {"type": "number"},
                },
                "required": ["label"],
                "additionalProperties": False,
            },
        },
    },
)
data = json.loads(resp.choices[0].message.content)   # parses, guaranteed

TypeScript

const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Classify: 'refund my order'" }],
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "intent",
      schema: {
        type: "object",
        properties: {
          label: { type: "string", enum: ["refund", "billing", "other"] },
          confidence: { type: "number" },
        },
        required: ["label"],
        additionalProperties: false,
      },
    },
  },
});

{"type": "json_object"} (JSON mode) is also supported: the content is guaranteed to parse as JSON, with no schema check. A malformed response_format returns a 400 with a specific message — it is never silently ignored.

Notes:

  • Nested objects, arrays, enum, required, and additionalProperties: false are all enforced. Schemas do not need to satisfy OpenAI's strict-mode restrictions (optional fields are fine).
  • The root schema must declare "type": "object" (same as OpenAI). Local $ref/$defs are supported; external $ref URLs are rejected with a 400. Schemas are capped at 50KB.
  • The format keyword (date-time, email, …) is not enforced — matching OpenAI, which also ignores format in structured outputs. Use pattern or enum when the shape matters.
  • With response_format set, temperature is not applied (structured serving runs at its canonical settings), and streamed responses deliver the JSON as a single buffered chunk after validation rather than token-by-token.
  • Clarifying questions are suppressed: a structured request always answers with schema-conformant JSON, never a prose question.

Tool calling (tools)​

model="auto" supports OpenAI-compatible function calling. Pass a tools array and the response comes back with tool_calls whose arguments are valid JSON for the schema you declared. Pareta serves tool turns on open models trained for tool use, and escalates when the turn needs more capability than they have.

You drive the loop. One request is one turn: Pareta returns either a final message or the tool calls it wants made. You execute them, append the results as role: "tool" messages, and call again.

Python

tools = [{
    "type": "function",
    "function": {
        "name": "get_stock_price",
        "description": "Get the current trading price for a ticker symbol.",
        "parameters": {
            "type": "object",
            "properties": {"symbol": {"type": "string"}},
            "required": ["symbol"],
        },
    },
}]

messages = [{"role": "user", "content": "What is NVDA trading at?"}]
resp = pa.chat.completions.create(model="auto", messages=messages, tools=tools)

call = resp.choices[0].message.tool_calls[0]
args = json.loads(call.function.arguments)          # parses, guaranteed
result = get_stock_price(**args)                    # your function

messages += [
    {"role": "assistant", "tool_calls": resp.choices[0].message.tool_calls},
    {"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)},
]
final = pa.chat.completions.create(model="auto", messages=messages, tools=tools)
print(final.choices[0].message.content)

TypeScript

const tools = [{
  type: "function",
  function: {
    name: "get_stock_price",
    description: "Get the current trading price for a ticker symbol.",
    parameters: {
      type: "object",
      properties: { symbol: { type: "string" } },
      required: ["symbol"],
    },
  },
}];

let messages = [{ role: "user", content: "What is NVDA trading at?" }];
const resp = await pa.chat.completions.create({ model: "auto", messages, tools });

const call = resp.choices[0].message.tool_calls[0];
const args = JSON.parse(call.function.arguments);
const result = await getStockPrice(args.symbol);

messages = [
  ...messages,
  { role: "assistant", tool_calls: resp.choices[0].message.tool_calls },
  { role: "tool", tool_call_id: call.id, content: JSON.stringify(result) },
];
const final = await pa.chat.completions.create({ model: "auto", messages, tools });

tool_choice works as it does on OpenAI: "auto" (the default) lets the model decide, "required" forces some tool call, "none" forbids them, and {"type": "function", "function": {"name": "..."}} forces a specific one. parallel_tool_calls is passed through.

Notes:

  • tools and response_format cannot be combined — the request returns a 400. The two guarantees are enforced by different machinery, and accepting both would silently drop the schema guarantee rather than honor it.
  • stream=True with tools returns a 400 today. Retry without it.
  • n > 1 returns a 400.
  • If the serving member is cold, you get a 503 with a Retry-After header rather than a hung request. Retry the call.
  • Billing is per turn, and every response carries the usual X-Pareta-Billed and X-Pareta-Frontier-Would-Have-Cost headers, so a multi-turn tool loop shows its cost turn by turn.

Streaming​

Set stream=True and create() returns an iterator of ChatCompletionChunk objects instead of a single ChatCompletion. Each chunk carries a delta (not a message); the incremental text is at chunk.choices[0].delta.content.

Python

with Pareta.from_env() as pa:
    stream = pa.chat.completions.create(
        model="auto",
        messages=[{"role": "user", "content": "Draft a one-paragraph status update."}],
        stream=True,
    )
    for chunk in stream:
        print(chunk.choices[0].delta.content or "", end="", flush=True)
    print()

TypeScript

const pa = Pareta.fromEnv();
const stream = pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Draft a one-paragraph status update." }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0].delta.content || "");
}
console.log();

ChatCompletionChunk has the same schema as ChatCompletion; it exists as a distinct type only for hinting. Guard delta.content with or "": the first and last chunks of a stream often carry role or finish metadata with no text.

The stream is data-only SSE and always terminates on a [DONE] sentinel, which the SDK consumes for you, so the iterator simply ends. Note that retries only cover the initial handshake. Once tokens are flowing, a mid-stream drop raises immediately rather than silently resuming.

Async​

AsyncPareta mirrors the sync client. Methods are async def; for streaming you await the call once, then async for over the chunks.

Python

import asyncio
from pareta import AsyncPareta

async def main():
    async with AsyncPareta.from_env() as pa:
        # Non-streaming
        resp = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "What is the invoice total?"}],
        )
        print(resp.choices[0].message.content)

        # Streaming
        stream = await pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Stream me a haiku about ledgers."}],
            stream=True,
        )
        async for chunk in stream:
            print(chunk.choices[0].delta.content or "", end="", flush=True)
        print()

asyncio.run(main())

TypeScript

// There is no AsyncPareta in TypeScript — the one Pareta client is already
// Promise-only. Every I/O method returns a Promise you `await`; streaming
// returns an AsyncIterable you drive with `for await`.
import { Pareta } from "pareta";

const pa = Pareta.fromEnv();

// Non-streaming
const resp = await pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "What is the invoice total?" }],
});
console.log(resp.choices[0].message.content);

// Streaming
const stream = pa.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Stream me a haiku about ledgers." }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0].delta.content || "");
}
console.log();

Reading the cost of a request​

Every completion tells you what it cost without any SDK: the X-Pareta-Billed response header is the debit in micro-USD, and X-Pareta-Frontier-Would-Have-Cost is what a single list-priced frontier call on the same prompt would have cost — each response carries its own savings receipt. Streamed responses deliver the same two numbers as SSE comment lines just before [DONE]. See the HTTP API reference for details.

Handling metering and not-ready errors​

Two error cases are specific to running inference. Both subclass ParetaError, so a single except ParetaError is a fine catch-all; the specific classes let you branch.

Python

from pareta import (
    Pareta,
    InsufficientCreditsError,   # 402: org balance empty
    EndpointNotReadyError,      # 503: a serving backend is warming / briefly unavailable
)

with Pareta.from_env() as pa:
    try:
        resp = pa.chat.completions.create(
            model="auto",
            messages=[{"role": "user", "content": "Hello"}],
        )
        print(resp.choices[0].message.content)
    except InsufficientCreditsError:
        # Balance hit zero. Top up in the dashboard (billing is browser-only);
        # the SDK exposes no balance or payment surface.
        print("Out of credit. Top up in the dashboard, then retry.")
    except EndpointNotReadyError:
        # A serving backend behind auto is warming. The SDK already retried
        # the 503 with backoff; wait briefly and retry the call.
        print("Backend warming — retry shortly.")

TypeScript

import {
  Pareta,
  InsufficientCreditsError,   // 402: org balance empty
  EndpointNotReadyError,      // 503: a serving backend is warming / briefly unavailable
} from "pareta";

const pa = Pareta.fromEnv();
try {
  const resp = await pa.chat.completions.create({
    model: "auto",
    messages: [{ role: "user", content: "Hello" }],
  });
  console.log(resp.choices[0].message.content);
} catch (e) {
  if (e instanceof InsufficientCreditsError) {
    // Balance hit zero. Top up in the dashboard (billing is browser-only);
    // the SDK exposes no balance or payment surface.
    console.log("Out of credit. Top up in the dashboard, then retry.");
  } else if (e instanceof EndpointNotReadyError) {
    // A serving backend behind auto is warming. The SDK already retried
    // the 503 with backoff; wait briefly and retry the call.
    console.log("Backend warming — retry shortly.");
  } else {
    throw e;
  }
}

Transient failures (429 rate limits, 5xx, connection timeouts) are retried automatically with exponential backoff, max_retries times (default 2). You only see RateLimitError or APITimeoutError after retries are exhausted. See Errors for the full hierarchy.

Using the OpenAI SDK instead​

Because Pareta is one OpenAI-compatible endpoint, you don't need this SDK to call it. Point the openai client at Pareta's base URL with your pareta_sk_ key. Note the /v1 suffix the OpenAI client expects:

Python

from openai import OpenAI

client = OpenAI(api_key="pareta_sk_...", base_url="https://api.pareta.ai/v1")

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What is the invoice total?"}],
)
print(resp.choices[0].message.content)

TypeScript

import OpenAI from "openai";

const client = new OpenAI({ apiKey: "pareta_sk_...", baseURL: "https://api.pareta.ai/v1" });

const resp = await client.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "What is the invoice total?" }],
});
console.log(resp.choices[0].message.content);

Tooling that discovers model ids by listing keeps working too: models.list() (GET /v1/models) returns exactly one entry, "auto" — there is only one model id to call. Field details in the models reference.

Streaming, temperature, max_tokens, and the rest work exactly as they do against OpenAI. Metering still applies: a zero balance returns a 402, which the openai client surfaces as its own status error. Reach for the Pareta SDK when you want typed errors and the control plane: running evals on your own data and reading auto's metrics (core concepts).