API protocol

Gemini Omni: parameters, scenarios and API guide

Preview guide

Paste into an AI assistant to help with integration. Documentation and examples only; no real credentials.

Preview guide

Gemini Omni: parameters, scenarios and API guide

Direct answerUse prompt or input, not both. input accepts text or a content array. input_reference accepts JPG/PNG/WebP images, not videos or task IDs.

Updated · Reviewed

Purpose and workflows

Video generation and conversational editing use distinct image, video and continuation fields. Parsed input does not prove successful generation.

This guide checks the site request contract; it does not claim paid generation acceptance for every scenario. Use the exact model, group and available endpoint shown in the marketplace. Replace example.com media and placeholder task IDs. Confirm a minimal request before adding references and advanced parameters. Consult marketplace pricing and actual billing for charges.

Models and endpoints

gemini-omni-1.1-flash, gemini-omni-1.2-flash

POST /v1/videos

Reference media formats

MediaFieldsEncoding and transportPreparation and limits
Image referencesinput_reference 或 input[] (type=image)input_reference accepts only JPG/PNG/WebP HTTP(S) image URLs. An input image item uses uri or mime_type plus raw Base64 data, exclusively.data has no data:image/...;base64, prefix. Use image/jpeg, image/png or image/webp. Image URL support does not imply video URL support.
Reference / edit videoinput[]: type=video, mime_type=video/mp4, dataThis example uses raw Base64 in data without the data:video/mp4;base64,... prefix. The adapter also forwards uri, but a prior video-URL test failed generation; this example is also unverified.Read local MP4 bytes and encode them. If you only have a URL, download it in your application first. Use prompt or input, not both. Larger inline videos currently fail task persistence; general availability is not accepted.
Reference audio—This video endpoint accepts only text/image/video input; audio, reference_audio and voice_id are not exposed.Generating a video with audio does not imply support for separate reference-audio input.
Task / asset IDsprevious_interaction_id + generation_config.video_config.taskUse an accessible successful site task_ ID, not a video URL or input[].data.edit/extend continuation is separate from reference-video upload. The recorded edit 400 remains unresolved; a Base64 reference test does not validate editing.

Parameters and combinations

  • Use prompt or input, not both. input accepts text or a content array. input_reference accepts JPG/PNG/WebP images, not videos or task IDs.
  • 1.1 accepts 3–10 seconds and 1.2 accepts 3–15. Continuations/extensions allow at most 10. Defaults are 10 seconds and 720p. Do not send watermark.
  • The reference-video example uses input[].type=video, mime_type=video/mp4 and raw Base64 data. The adapter also accepts video uri, but a prior URL test failed during upstream generation. Neither form has verified generation support.
  • Use a completed accessible site task_ ID in previous_interaction_id, not input_reference. Put edit/extend in generation_config.video_config.task.
  • The adapter accepts uri or raw Base64 data for both images and videos; video examples use data, without verified generation support. Omit the data-URL prefix and set mime_type separately. The site has no general Google Files upload endpoint.
  • 2026-11-15 business-site 1.1 Flash control: three-second text generation completed. Public video input and previous_interaction_id with task=edit were accepted but failed with Synchronous video request failed (HTTP 400). Version 1.2 and extend were not tested in this round.
  • 2026-11-15 Base64 control: business-site 1.1 accepted a small MP4 but generation still failed (HTTP 400); a larger inline video failed task persistence. Correcting the transport does not establish reference-video acceptance. Version 1.2 remains untested.
  • Official 1.2: ≤6 reference images; outputs of 3–10s allow ≤3 reference clips of ≤3s each, 11–15s allow one. Editing input ≤15s; extension adds 3–10s up to 30s total. Site transport and acceptance limits above still apply.
  • Officially no audio references, voice editing, system instructions or separate negative-prompt field. Outputs contain invisible SynthID watermarking without a disable switch.
FieldTypeUse and constraints
Authorizationheader stringRequired: Bearer <site API key>, with Content-Type: application/json.
modelstringRequired: gemini-omni-1.1-flash or gemini-omni-1.2-flash.
promptstringSite text shortcut, mutually exclusive with input; may pair with input_reference.
inputstring / object[]Official text or structured multimodal input; do not duplicate with prompt/input_reference on this service.
input[].typestringRequired text/image/video; standalone audio input is unsupported.
input[].textstringRequired for text. Describe first/last frames, references and edits in natural language; omit deprecated control tags such as <FIRST_FRAME>.
input[].mime_typestringRequired for images/videos, matching the bytes, e.g. image/png, image/jpeg or video/mp4.
input[].uristringThe adapter forwards public HTTP(S) URIs for both images and videos. A prior 1.1 video-URI request was accepted but failed during upstream generation; forwarding does not establish generation support. Version 1.2 is unverified.
input[].datastringRaw Base64 bytes without a data-URI prefix; mutually exclusive with uri. Used in the video example, without verified generation support; see conditional examples for known persistence/generation failures.
input_referencestringSite single-image shortcut with prompt. URL must identify JPG/PNG/WebP; otherwise use input with explicit mime_type.
secondsinteger / stringSite duration alias, default 10: 1.1 supports 3–10s, ordinary 1.2 generation 3–15s. First/last frames and site task continuation remain ≤10s.
sizestringSite pixel-size alias: landscape/portrait 360p/720p/1080p/4k; default 1280x720. Avoid duplicating corresponding response_format settings.
response_format.typestringFixed to video on the site; supplied by the adapter.
response_format.durationstringFor example 3s or 8s; excludes seconds and follows the same model/mode limits.
response_format.resolutionstring360p/720p/1080p/4k, default 720p; do not duplicate with size.
response_format.aspect_ratiostring16:9 (default) or 9:16.
response_format.deliverystringSite fixes delivery to uri and returns the successful task video URL. Official inline Base64 output is not an exposed site option.
response_format.fpsintegerSite compatibility field, only 24; frame rate is fixed. Not an official response_format field and removed before forwarding.
generation_config.video_config.taskstringOptional text_to_video/image_to_video/reference_to_video/edit/extend. Prefer natural-language selection; explicit tasks add stricter media constraints.
generation_config.temperature / generation_config.top_pnumberExplicitly unsupported by official Omni; omit even if the site object accepts them.
generation_config.seed / generation_config.max_output_tokensintegerGeneric Interactions options forwarded by the site; the current Omni guide does not confirm video behavior/ranges. Not promised as video controls and removed from ordinary examples.
previous_interaction_idstringUse an accessible successful site task_ ID on the same route with compatible credentials; mapped to an official interaction ID. Each continuation ≤10s; not a video URL.
background / stream / storeofficial controlsSite asynchronous video only accepts background=true and stream=false; execution is route-managed. store is not exposed. Do not copy other official Interactions execution modes.

Scenarios and code examples

Prepare a Base64 request from local MP4 (JSON only)

Either script reads local action.mp4 and writes request.json without submitting or billing. Download a remote video locally first; encoding the URL text is not encoding video bytes. MIME must match the file. Larger Base64 inputs currently fail persistence, and a small generation sample still returned 400. Do not automatically retry POST. The 1 MiB encoded-size check is only a request ceiling, not a guarantee of success below it.

import base64
import json
from pathlib import Path

# Use a small local MP4. This prepares JSON; it does not submit a paid task.
video = Path("action.mp4").read_bytes()
if not video:
    raise ValueError("Empty video")
request = {
    "model": "gemini-omni-1.1-flash",  # Selected marketplace model
    "input": [
        {"type": "text", "text": "参考视频动作,创作新的花园场景。"},
        {"type": "video", "mime_type": "video/mp4",
         "data": base64.b64encode(video).decode("ascii")},
    ],
    "seconds": "3",
    "size": "640x360",
}
encoded = json.dumps(request, ensure_ascii=False).encode("utf-8")
# Current request-path ceiling, NOT a guarantee that smaller inputs will succeed.
if len(encoded) > 1024 * 1024:
    raise ValueError("Encoded JSON exceeds the current 1 MiB request-path ceiling")
with Path("request.json").open("xb") as output:
    output.write(encoded)
# No top-level prompt, no uri, no data:video/mp4;base64, prefix.

import { readFileSync, writeFileSync } from 'node:fs';

// Prepare JSON only; this does not submit a paid task.
const video = readFileSync('action.mp4');
if (!video.length) throw new Error('Empty video');
const request = {
  model: 'gemini-omni-1.1-flash', // Selected marketplace model
  input: [
    { type: 'text', text: '参考视频动作,创作新的花园场景。' },
    { type: 'video', mime_type: 'video/mp4', data: video.toString('base64') },
  ],
  seconds: '3',
  size: '640x360',
};
const encoded = JSON.stringify(request);
// Current request-path ceiling, NOT a guarantee of successful generation.
if (Buffer.byteLength(encoded) > 1024 * 1024) {
  throw new Error('Encoded JSON exceeds the current 1 MiB request-path ceiling');
}
writeFileSync('request.json', encoded, { flag: 'wx' });
// No top-level prompt, no uri, no data:video/mp4;base64, prefix.

For an authorized test, pass request.json to the overview submission/polling/download script. Submit once, then query the site task ID.

Text to video

Applies to: gemini-omni-1.1-flash, gemini-omni-1.2-flash

POST /v1/videos

{
  "model": "gemini-omni-1.2-flash",
  "prompt": "一颗玻璃球沿木制轨道滚动,连续镜头。",
  "seconds": "8",
  "size": "1280x720"
}

curl

curl --fail-with-body --silent --show-error --connect-timeout 10 --max-time 120 --request POST 'https://api.tu-zi.com/v1/videos' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
  "model": "gemini-omni-1.2-flash",
  "prompt": "一颗玻璃球沿木制轨道滚动,连续镜头。",
  "seconds": "8",
  "size": "1280x720"
}'

python

import json
import os
import requests

url = "https://api.tu-zi.com/v1/videos"
headers = json.loads("{\"Authorization\":\"Bearer YOUR_API_KEY\",\"Content-Type\":\"application/json\"}")
for name in ["API_KEY"]:
    if not os.environ.get(name, "").strip():
        raise RuntimeError("Set environment variable: " + name)
headers["Authorization"] = "Bearer " + os.environ["API_KEY"]
payload = json.loads("{\"model\":\"gemini-omni-1.2-flash\",\"prompt\":\"一颗玻璃球沿木制轨道滚动,连续镜头。\",\"seconds\":\"8\",\"size\":\"1280x720\"}")
response = requests.request("POST", url, headers=headers, json=payload, timeout=120)
response.raise_for_status()
result = response.json()
if isinstance(result, dict) and (result.get("success") is False or result.get("code") is False or result.get("error")):
    raise RuntimeError(json.dumps(result, ensure_ascii=False))
print(json.dumps(result, ensure_ascii=False, indent=2))

javascript

// Node.js 20+
const url = "https://api.tu-zi.com/v1/videos";
const headers = {
  "Authorization": "Bearer YOUR_API_KEY",
  "Content-Type": "application/json"
};
for (const name of ["API_KEY"]) {
  if (!process.env[name]?.trim()) throw new Error("Set environment variable: " + name);
}
headers.Authorization = 'Bearer ' + process.env.API_KEY;
const response = await fetch(url, { method: "POST", headers, body: JSON.stringify({
  "model": "gemini-omni-1.2-flash",
  "prompt": "一颗玻璃球沿木制轨道滚动,连续镜头。",
  "seconds": "8",
  "size": "1280x720"
}), signal: AbortSignal.timeout(120000) });
if (!response.ok) throw new Error(await response.text());
const result = await response.json();
if (result?.success === false || result?.code === false || result?.error) throw new Error(JSON.stringify(result));
console.log(JSON.stringify(result, null, 2));

Single image reference

Applies to: gemini-omni-1.1-flash, gemini-omni-1.2-flash

POST /v1/videos

{
  "model": "gemini-omni-1.2-flash",
  "prompt": "保持图片中的人物外观,让人物向镜头挥手。",
  "input_reference": "https://example.com/subject.png",
  "seconds": "8",
  "size": "720x1280"
}

curl

curl --fail-with-body --silent --show-error --connect-timeout 10 --max-time 120 --request POST 'https://api.tu-zi.com/v1/videos' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
  "model": "gemini-omni-1.2-flash",
  "prompt": "保持图片中的人物外观,让人物向镜头挥手。",
  "input_reference": "https://example.com/subject.png",
  "seconds": "8",
  "size": "720x1280"
}'

python

import json
import os
import requests

url = "https://api.tu-zi.com/v1/videos"
headers = json.loads("{\"Authorization\":\"Bearer YOUR_API_KEY\",\"Content-Type\":\"application/json\"}")
for name in ["API_KEY"]:
    if not os.environ.get(name, "").strip():
        raise RuntimeError("Set environment variable: " + name)
headers["Authorization"] = "Bearer " + os.environ["API_KEY"]
payload = json.loads("{\"model\":\"gemini-omni-1.2-flash\",\"prompt\":\"保持图片中的人物外观,让人物向镜头挥手。\",\"input_reference\":\"https://example.com/subject.png\",\"seconds\":\"8\",\"size\":\"720x1280\"}")
response = requests.request("POST", url, headers=headers, json=payload, timeout=120)
response.raise_for_status()
result = response.json()
if isinstance(result, dict) and (result.get("success") is False or result.get("code") is False or result.get("error")):
    raise RuntimeError(json.dumps(result, ensure_ascii=False))
print(json.dumps(result, ensure_ascii=False, indent=2))

javascript

// Node.js 20+
const url = "https://api.tu-zi.com/v1/videos";
const headers = {
  "Authorization": "Bearer YOUR_API_KEY",
  "Content-Type": "application/json"
};
for (const name of ["API_KEY"]) {
  if (!process.env[name]?.trim()) throw new Error("Set environment variable: " + name);
}
headers.Authorization = 'Bearer ' + process.env.API_KEY;
const response = await fetch(url, { method: "POST", headers, body: JSON.stringify({
  "model": "gemini-omni-1.2-flash",
  "prompt": "保持图片中的人物外观,让人物向镜头挥手。",
  "input_reference": "https://example.com/subject.png",
  "seconds": "8",
  "size": "720x1280"
}), signal: AbortSignal.timeout(120000) });
if (!response.ok) throw new Error(await response.text());
const result = await response.json();
if (result?.success === false || result?.code === false || result?.error) throw new Error(JSON.stringify(result));
console.log(JSON.stringify(result, null, 2));

Multiple images and output format

Applies to: gemini-omni-1.1-flash, gemini-omni-1.2-flash

POST /v1/videos

{
  "model": "gemini-omni-1.2-flash",
  "input": [
    {
      "type": "text",
      "text": "人物在第二张图的场景中行走。"
    },
    {
      "type": "image",
      "mime_type": "image/png",
      "uri": "https://example.com/subject.png"
    },
    {
      "type": "image",
      "mime_type": "image/png",
      "uri": "https://example.com/end.png"
    }
  ],
  "response_format": {
    "type": "video",
    "duration": "8s",
    "resolution": "720p",
    "aspect_ratio": "16:9",
    "delivery": "uri"
  }
}

curl

curl --fail-with-body --silent --show-error --connect-timeout 10 --max-time 120 --request POST 'https://api.tu-zi.com/v1/videos' \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
  "model": "gemini-omni-1.2-flash",
  "input": [
    {
      "type": "text",
      "text": "人物在第二张图的场景中行走。"
    },
    {
      "type": "image",
      "mime_type": "image/png",
      "uri": "https://example.com/subject.png"
    },
    {
      "type": "image",
      "mime_type": "image/png",
      "uri": "https://example.com/end.png"
    }
  ],
  "response_format": {
    "type": "video",
    "duration": "8s",
    "resolution": "720p",
    "aspect_ratio": "16:9",
    "delivery": "uri"
  }
}'

python

import json
import os
import requests

url = "https://api.tu-zi.com/v1/videos"
headers = json.loads("{\"Authorization\":\"Bearer YOUR_API_KEY\",\"Content-Type\":\"application/json\"}")
for name in ["API_KEY"]:
    if not os.environ.get(name, "").strip():
        raise RuntimeError("Set environment variable: " + name)
headers["Authorization"] = "Bearer " + os.environ["API_KEY"]
payload = json.loads("{\"model\":\"gemini-omni-1.2-flash\",\"input\":[{\"type\":\"text\",\"text\":\"人物在第二张图的场景中行走。\"},{\"type\":\"image\",\"mime_type\":\"image/png\",\"uri\":\"https://example.com/subject.png\"},{\"type\":\"image\",\"mime_type\":\"image/png\",\"uri\":\"https://example.com/end.png\"}],\"response_format\":{\"type\":\"video\",\"duration\":\"8s\",\"resolution\":\"720p\",\"aspect_ratio\":\"16:9\",\"delivery\":\"uri\"}}")
response = requests.request("POST", url, headers=headers, json=payload, timeout=120)
response.raise_for_status()
result = response.json()
if isinstance(result, dict) and (result.get("success") is False or result.get("code") is False or result.get("error")):
    raise RuntimeError(json.dumps(result, ensure_ascii=False))
print(json.dumps(result, ensure_ascii=False, indent=2))

javascript

// Node.js 20+
const url = "https://api.tu-zi.com/v1/videos";
const headers = {
  "Authorization": "Bearer YOUR_API_KEY",
  "Content-Type": "application/json"
};
for (const name of ["API_KEY"]) {
  if (!process.env[name]?.trim()) throw new Error("Set environment variable: " + name);
}
headers.Authorization = 'Bearer ' + process.env.API_KEY;
const response = await fetch(url, { method: "POST", headers, body: JSON.stringify({
  "model": "gemini-omni-1.2-flash",
  "input": [
    {
      "type": "text",
      "text": "人物在第二张图的场景中行走。"
    },
    {
      "type": "image",
      "mime_type": "image/png",
      "uri": "https://example.com/subject.png"
    },
    {
      "type": "image",
      "mime_type": "image/png",
      "uri": "https://example.com/end.png"
    }
  ],
  "response_format": {
    "type": "video",
    "duration": "8s",
    "resolution": "720p",
    "aspect_ratio": "16:9",
    "delivery": "uri"
  }
}), signal: AbortSignal.timeout(120000) });
if (!response.ok) throw new Error(await response.text());
const result = await response.json();
if (result?.success === false || result?.code === false || result?.error) throw new Error(JSON.stringify(result));
console.log(JSON.stringify(result, null, 2));

Base64 video reference

Applies to: gemini-omni-1.1-flash, gemini-omni-1.2-flash

POST /v1/videos

Encode MP4 as raw Base64 and replace the data placeholder; public video URLs are unsupported. On 2026-11-15, a small business-site 1.1 sample failed generation with HTTP 400, and a larger input failed task persistence. Version 1.2 is untested; reference-video generation is not accepted.

{
  "model": "gemini-omni-1.2-flash",
  "input": [
    {
      "type": "text",
      "text": "参考片段中的动作,创作新的花园场景。"
    },
    {
      "type": "video",
      "mime_type": "video/mp4",
      "data": "REPLACE_WITH_RAW_BASE64_MP4"
    }
  ],
  "seconds": "8",
  "size": "1280x720"
}

Edit a previous task (conditional)

Applies to: gemini-omni-1.1-flash, gemini-omni-1.2-flash

POST /v1/videos

Requires a completed continuable site task. On 2026-11-15, business-site 1.1 Flash failed with HTTP 400 despite correct fields and a fresh successful source task. Version 1.2 is untested; editing is not accepted.

{
  "model": "gemini-omni-1.2-flash",
  "prompt": "保留人物动作,把天空改为夕阳。",
  "previous_interaction_id": "task_REPLACE_WITH_SUCCESSFUL_VIDEO",
  "seconds": "8",
  "size": "1280x720",
  "generation_config": {
    "video_config": {
      "task": "edit"
    }
  }
}

Extend a previous task (conditional)

Applies to: gemini-omni-1.1-flash, gemini-omni-1.2-flash

POST /v1/videos

Continuation is limited to 10 seconds and requires eligible task access; output is unverified.

{
  "model": "gemini-omni-1.2-flash",
  "prompt": "继续上一段镜头,人物走向远处。",
  "previous_interaction_id": "task_REPLACE_WITH_SUCCESSFUL_VIDEO",
  "seconds": "8",
  "generation_config": {
    "video_config": {
      "task": "extend"
    }
  }
}

Task results and downloads

Save id or task_id from submission. Poll GET /v1/videos/{task_id}: queued / in_progress lead to completed or failed. Read video_url on completion (some responses also expose metadata.url). Resolve relative URLs against the API origin. The site /content endpoint uses your site key; external downloads must not receive that key. Download expiring results promptly. Resume GET after a timeout rather than automatically repeating POST.

Complete unified video Python / JavaScript polling and download examples

Troubleshooting

SymptomCheck
Image validation / URI errorEnsure a video or task ID was not placed in an image field; check the URL returns media
Unsupported content type / fieldCheck the exact family and content/contents/input schema; voice_id cannot be an audio URL
400 conflicting fieldsCheck prompt/input, frames/references, and seconds/response_format exclusivity
HTTP success but task failedRetain the site task ID, time, model, domain and redacted request; submission 200 is not output success
404 or inaccessible taskCheck domain, key access and task type; do not use IDs from another site or the official API
Local file cannot be fetchedUse a supported upload flow or a reachable HTTPS media URL

Sources and scope

Gemini Omni

Use cases

  • Choose video models and media formats
  • Diagnose request and task failures

API protocols

  • /v1/videos

FAQ

Does a task ID prove generation succeeded?

No. Save the site task ID, poll to a successful terminal state and verify the result. Inspect failures instead of automatically creating a new task.

Can I put a local image, video or WAV path in JSON?

No. URL fields require reachable media, not localhost, disk paths or browser-login links. Kling PCM WAV requires an enabled multipart voice endpoint (both sites returned 404 on 2026-11-15). Follow each family’s asset workflow for other media.

Can parameters and task IDs be reused across models?

Do not assume compatibility. Check the exact model, endpoint, casing, media role and exclusive fields. Editing, drafts and voices require accessible successful tasks of the correct kind. Official capabilities are not evidence of site acceptance.

Official sources

  1. Gemini Omni Official