StepFun | StepAudio 3 Realtime
EN 中文
API Product page

StepAudio 3 Realtime

Evaluation & Method

A clear view of the request configuration, benchmark protocols, scoring conventions, and text-based evaluation used for the StepAudio 3 Realtime audio-understanding results.

Audio in · Text out stepaudio-3-chat-preview temperature 0.7 max_tokens 32000

Request configuration

One request shape.
Every benchmark.

The reported audio-understanding results use the public Chat Completions interface. Benchmark-specific prompts and context are assembled around the same model request.

POST Endpoint

api.stepfun.com/v1/chat/completions

S3 Realtime Model

stepaudio-3-chat-preview

0.7 Temperature

The same generation temperature is used across the benchmark requests.

32,000 Max tokens

The output budget covers the model's reasoning and final text response.

Audio input

Audio is sent as a base64-encoded input_audio content block. The benchmark runner declares the audio format explicitly and applies the task-specific audio preparation required by each dataset.

01 Audio Waveform input
02 Context Prompt and history
03 Model Audio-language response
04 Text Final answer
05 Score Metric or judge
chat-completions.json 
{
  "model": "stepaudio-3-chat-preview",
  "temperature": 0.7,
  "max_tokens": 32000,
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "Please describe this audio."},
      {
        "type": "input_audio",
        "input_audio": {
          "data": "<base64-audio>",
          "format": "wav"
        }
      }
    ]
  }]
}

Benchmark protocols

Eight benchmarks.
One readable map.

Each benchmark keeps its own task structure and primary metric. The table below summarizes the input/output mode, context, judge, and reported score.

Benchmark Task Context Judge Primary metric
MMSU Audio multiple-choice Single turn gpt-oss-120b Accuracy
MMAR Audio reasoning multiple-choice Single turn gpt-oss-120b Accuracy
MMAU mini Audio multiple-choice Single turn gpt-oss-120b Accuracy
Big Bench Audio Audio-grounded reasoning Single turn gpt-oss-120b Accuracy
Audio Multi-Challenge Multi-turn interaction with rubrics Multi-turn o4-mini APR / ARS
WildSpeech-Bench Long-audio open-ended questions Question in audio gpt-oss-120b Mean score on 0–100 scale
Step-Caption Structured audio description Single turn gpt-oss-120b Overall score
MTalk-Bench Multi-turn spoken dialogue in S2T format Multi-turn gpt-oss-120b Macro-averaged pass rate

Scoring and adaptation

Keep the task intact.
Make the output comparable.

The benchmark answer keys and rubrics remain the basis for scoring. Adaptations are limited to the audio-to-text interface, multi-turn context, and output format needed by the public Chat Completions endpoint.

Text-only LLM judges

Open-ended responses are evaluated with a text-only judge. The judge receives the task text, answer key or rubric, and the model's final text response; it does not receive the audio. The general text-based tasks use gpt-oss-120b, while Audio Multi-Challenge uses o4-mini. Where supported, judge requests use temperature 0.8, top_p 0.9, frequency penalty 0.1, and presence penalty 0.1. Empty, failed, or unparsable judge responses are retried before aggregation.

Metric aggregation

Each benchmark keeps its native primary metric. For rubric tasks, rubric items are averaged within a case before cases are averaged with equal weight. If a cross-benchmark summary is needed, metrics are first expressed on a 0–100 scale and then macro-averaged without sample-count weighting. Multiple-choice tasks report accuracy, Audio Multi-Challenge reports APR and ARS, WildSpeech-Bench maps its 1–10 judge score to 0–100, and MTalk-Bench reports a case-level macro pass rate.

Dataset-specific method

Where the request
meets the benchmark.

The following adaptations make the public audio-to-text interface usable while keeping the benchmark's task evidence and scoring structure visible.

Benchmark family Request construction Scoring treatment
MMSU / MMAR / MMAU mini Official question and options are sent with an output-format instruction and audio input. Answer correctness is judged against the official answer structure.
Big Bench Audio The question is contained in the recording; audio is followed by the fixed answer instruction. Open-ended text answers are evaluated against the task answer.
Audio Multi-Challenge Multi-turn audio and text history are preserved for the final response. Official rubric structure with APR / ARS and the o4-mini judge.
WildSpeech-Bench The model receives audio and returns text directly instead of synthesizing a voice response. The checklist-based 1–10 score is mapped linearly to 0–100.
Step-Caption Audio is followed by a natural-language description instruction. The structured description dimensions are assessed by the text judge.
MTalk-Bench Current-turn audio is combined with official text history and previous model replies. Semantic, paralinguistic, and ambient subsets use the same S2T text-rubric process.

Resources

Read the details.
Try the model.

The product page remains the home for demos and headline results. These links provide the public API, report, and evaluation references.

Realtime product page

Explore the model story, demos, and headline benchmark results.

Open product page ↗

Chat Completions API

Read the public API reference for audio input and text responses.

Open API reference ↗

Technical report

Read the architecture, training approach, and complete reported results.

Read the report ↗