API protocol
Gemini Omni: parameters, scenarios and API guide
Preview guide
Paste into an AI assistant to help with integration. Documentation and examples only; no real credentials.
Direct answerUse prompt or input, not both. input accepts text or a content array. input_reference accepts JPG/PNG/WebP images, not videos or task IDs.
Updated · Reviewed
Purpose and workflows
Video generation and conversational editing use distinct image, video and continuation fields. Parsed input does not prove successful generation.
This guide checks the site request contract; it does not claim paid generation acceptance for every scenario. Use the exact model, group and available endpoint shown in the marketplace. Replace example.com media and placeholder task IDs. Confirm a minimal request before adding references and advanced parameters. Consult marketplace pricing and actual billing for charges.
Models and endpoints
gemini-omni-1.1-flash, gemini-omni-1.2-flash
POST /v1/videos
Reference media formats
| Media | Fields | Encoding and transport | Preparation and limits |
|---|---|---|---|
| Image references | input_reference 或 input[] (type=image) | input_reference accepts only JPG/PNG/WebP HTTP(S) image URLs. An input image item uses uri or mime_type plus raw Base64 data, exclusively. | data has no data:image/...;base64, prefix. Use image/jpeg, image/png or image/webp. Image URL support does not imply video URL support. |
| Reference / edit video | input[]: type=video, mime_type=video/mp4, data | This example uses raw Base64 in data without the data:video/mp4;base64,... prefix. The adapter also forwards uri, but a prior video-URL test failed generation; this example is also unverified. | Read local MP4 bytes and encode them. If you only have a URL, download it in your application first. Use prompt or input, not both. Larger inline videos currently fail task persistence; general availability is not accepted. |
| Reference audio | — | This video endpoint accepts only text/image/video input; audio, reference_audio and voice_id are not exposed. | Generating a video with audio does not imply support for separate reference-audio input. |
| Task / asset IDs | previous_interaction_id + generation_config.video_config.task | Use an accessible successful site task_ ID, not a video URL or input[].data. | edit/extend continuation is separate from reference-video upload. The recorded edit 400 remains unresolved; a Base64 reference test does not validate editing. |
Parameters and combinations
- Use prompt or input, not both. input accepts text or a content array. input_reference accepts JPG/PNG/WebP images, not videos or task IDs.
- 1.1 accepts 3–10 seconds and 1.2 accepts 3–15. Continuations/extensions allow at most 10. Defaults are 10 seconds and 720p. Do not send watermark.
- The reference-video example uses input[].type=video, mime_type=video/mp4 and raw Base64 data. The adapter also accepts video uri, but a prior URL test failed during upstream generation. Neither form has verified generation support.
- Use a completed accessible site task_ ID in previous_interaction_id, not input_reference. Put edit/extend in generation_config.video_config.task.
- The adapter accepts uri or raw Base64 data for both images and videos; video examples use data, without verified generation support. Omit the data-URL prefix and set mime_type separately. The site has no general Google Files upload endpoint.
- 2026-11-15 business-site 1.1 Flash control: three-second text generation completed. Public video input and previous_interaction_id with task=edit were accepted but failed with Synchronous video request failed (HTTP 400). Version 1.2 and extend were not tested in this round.
- 2026-11-15 Base64 control: business-site 1.1 accepted a small MP4 but generation still failed (HTTP 400); a larger inline video failed task persistence. Correcting the transport does not establish reference-video acceptance. Version 1.2 remains untested.
- Official 1.2: ≤6 reference images; outputs of 3–10s allow ≤3 reference clips of ≤3s each, 11–15s allow one. Editing input ≤15s; extension adds 3–10s up to 30s total. Site transport and acceptance limits above still apply.
- Officially no audio references, voice editing, system instructions or separate negative-prompt field. Outputs contain invisible SynthID watermarking without a disable switch.
| Field | Type | Use and constraints |
|---|---|---|
Authorization | header string | Required: Bearer <site API key>, with Content-Type: application/json. |
model | string | Required: gemini-omni-1.1-flash or gemini-omni-1.2-flash. |
prompt | string | Site text shortcut, mutually exclusive with input; may pair with input_reference. |
input | string / object[] | Official text or structured multimodal input; do not duplicate with prompt/input_reference on this service. |
input[].type | string | Required text/image/video; standalone audio input is unsupported. |
input[].text | string | Required for text. Describe first/last frames, references and edits in natural language; omit deprecated control tags such as <FIRST_FRAME>. |
input[].mime_type | string | Required for images/videos, matching the bytes, e.g. image/png, image/jpeg or video/mp4. |
input[].uri | string | The adapter forwards public HTTP(S) URIs for both images and videos. A prior 1.1 video-URI request was accepted but failed during upstream generation; forwarding does not establish generation support. Version 1.2 is unverified. |
input[].data | string | Raw Base64 bytes without a data-URI prefix; mutually exclusive with uri. Used in the video example, without verified generation support; see conditional examples for known persistence/generation failures. |
input_reference | string | Site single-image shortcut with prompt. URL must identify JPG/PNG/WebP; otherwise use input with explicit mime_type. |
seconds | integer / string | Site duration alias, default 10: 1.1 supports 3–10s, ordinary 1.2 generation 3–15s. First/last frames and site task continuation remain ≤10s. |
size | string | Site pixel-size alias: landscape/portrait 360p/720p/1080p/4k; default 1280x720. Avoid duplicating corresponding response_format settings. |
response_format.type | string | Fixed to video on the site; supplied by the adapter. |
response_format.duration | string | For example 3s or 8s; excludes seconds and follows the same model/mode limits. |
response_format.resolution | string | 360p/720p/1080p/4k, default 720p; do not duplicate with size. |
response_format.aspect_ratio | string | 16:9 (default) or 9:16. |
response_format.delivery | string | Site fixes delivery to uri and returns the successful task video URL. Official inline Base64 output is not an exposed site option. |
response_format.fps | integer | Site compatibility field, only 24; frame rate is fixed. Not an official response_format field and removed before forwarding. |
generation_config.video_config.task | string | Optional text_to_video/image_to_video/reference_to_video/edit/extend. Prefer natural-language selection; explicit tasks add stricter media constraints. |
generation_config.temperature / generation_config.top_p | number | Explicitly unsupported by official Omni; omit even if the site object accepts them. |
generation_config.seed / generation_config.max_output_tokens | integer | Generic Interactions options forwarded by the site; the current Omni guide does not confirm video behavior/ranges. Not promised as video controls and removed from ordinary examples. |
previous_interaction_id | string | Use an accessible successful site task_ ID on the same route with compatible credentials; mapped to an official interaction ID. Each continuation ≤10s; not a video URL. |
background / stream / store | official controls | Site asynchronous video only accepts background=true and stream=false; execution is route-managed. store is not exposed. Do not copy other official Interactions execution modes. |
Scenarios and code examples
Prepare a Base64 request from local MP4 (JSON only)
Either script reads local action.mp4 and writes request.json without submitting or billing. Download a remote video locally first; encoding the URL text is not encoding video bytes. MIME must match the file. Larger Base64 inputs currently fail persistence, and a small generation sample still returned 400. Do not automatically retry POST. The 1 MiB encoded-size check is only a request ceiling, not a guarantee of success below it.
import base64
import json
from pathlib import Path
# Use a small local MP4. This prepares JSON; it does not submit a paid task.
video = Path("action.mp4").read_bytes()
if not video:
raise ValueError("Empty video")
request = {
"model": "gemini-omni-1.1-flash", # Selected marketplace model
"input": [
{"type": "text", "text": "参考视频动作,创作新的花园场景。"},
{"type": "video", "mime_type": "video/mp4",
"data": base64.b64encode(video).decode("ascii")},
],
"seconds": "3",
"size": "640x360",
}
encoded = json.dumps(request, ensure_ascii=False).encode("utf-8")
# Current request-path ceiling, NOT a guarantee that smaller inputs will succeed.
if len(encoded) > 1024 * 1024:
raise ValueError("Encoded JSON exceeds the current 1 MiB request-path ceiling")
with Path("request.json").open("xb") as output:
output.write(encoded)
# No top-level prompt, no uri, no data:video/mp4;base64, prefix.
import { readFileSync, writeFileSync } from 'node:fs';
// Prepare JSON only; this does not submit a paid task.
const video = readFileSync('action.mp4');
if (!video.length) throw new Error('Empty video');
const request = {
model: 'gemini-omni-1.1-flash', // Selected marketplace model
input: [
{ type: 'text', text: '参考视频动作,创作新的花园场景。' },
{ type: 'video', mime_type: 'video/mp4', data: video.toString('base64') },
],
seconds: '3',
size: '640x360',
};
const encoded = JSON.stringify(request);
// Current request-path ceiling, NOT a guarantee of successful generation.
if (Buffer.byteLength(encoded) > 1024 * 1024) {
throw new Error('Encoded JSON exceeds the current 1 MiB request-path ceiling');
}
writeFileSync('request.json', encoded, { flag: 'wx' });
// No top-level prompt, no uri, no data:video/mp4;base64, prefix.
For an authorized test, pass request.json to the overview submission/polling/download script. Submit once, then query the site task ID.
Text to video
Applies to: gemini-omni-1.1-flash, gemini-omni-1.2-flash
POST /v1/videos
{
"model": "gemini-omni-1.2-flash",
"prompt": "一颗玻璃球沿木制轨道滚动,连续镜头。",
"seconds": "8",
"size": "1280x720"
}
curl
curl --fail-with-body --silent --show-error --connect-timeout 10 --max-time 120 --request POST 'https://api.tu-zi.com/v1/videos' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-omni-1.2-flash",
"prompt": "一颗玻璃球沿木制轨道滚动,连续镜头。",
"seconds": "8",
"size": "1280x720"
}'
python
import json
import os
import requests
url = "https://api.tu-zi.com/v1/videos"
headers = json.loads("{\"Authorization\":\"Bearer YOUR_API_KEY\",\"Content-Type\":\"application/json\"}")
for name in ["API_KEY"]:
if not os.environ.get(name, "").strip():
raise RuntimeError("Set environment variable: " + name)
headers["Authorization"] = "Bearer " + os.environ["API_KEY"]
payload = json.loads("{\"model\":\"gemini-omni-1.2-flash\",\"prompt\":\"一颗玻璃球沿木制轨道滚动,连续镜头。\",\"seconds\":\"8\",\"size\":\"1280x720\"}")
response = requests.request("POST", url, headers=headers, json=payload, timeout=120)
response.raise_for_status()
result = response.json()
if isinstance(result, dict) and (result.get("success") is False or result.get("code") is False or result.get("error")):
raise RuntimeError(json.dumps(result, ensure_ascii=False))
print(json.dumps(result, ensure_ascii=False, indent=2))
javascript
// Node.js 20+
const url = "https://api.tu-zi.com/v1/videos";
const headers = {
"Authorization": "Bearer YOUR_API_KEY",
"Content-Type": "application/json"
};
for (const name of ["API_KEY"]) {
if (!process.env[name]?.trim()) throw new Error("Set environment variable: " + name);
}
headers.Authorization = 'Bearer ' + process.env.API_KEY;
const response = await fetch(url, { method: "POST", headers, body: JSON.stringify({
"model": "gemini-omni-1.2-flash",
"prompt": "一颗玻璃球沿木制轨道滚动,连续镜头。",
"seconds": "8",
"size": "1280x720"
}), signal: AbortSignal.timeout(120000) });
if (!response.ok) throw new Error(await response.text());
const result = await response.json();
if (result?.success === false || result?.code === false || result?.error) throw new Error(JSON.stringify(result));
console.log(JSON.stringify(result, null, 2));
Single image reference
Applies to: gemini-omni-1.1-flash, gemini-omni-1.2-flash
POST /v1/videos
{
"model": "gemini-omni-1.2-flash",
"prompt": "保持图片中的人物外观,让人物向镜头挥手。",
"input_reference": "https://example.com/subject.png",
"seconds": "8",
"size": "720x1280"
}
curl
curl --fail-with-body --silent --show-error --connect-timeout 10 --max-time 120 --request POST 'https://api.tu-zi.com/v1/videos' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-omni-1.2-flash",
"prompt": "保持图片中的人物外观,让人物向镜头挥手。",
"input_reference": "https://example.com/subject.png",
"seconds": "8",
"size": "720x1280"
}'
python
import json
import os
import requests
url = "https://api.tu-zi.com/v1/videos"
headers = json.loads("{\"Authorization\":\"Bearer YOUR_API_KEY\",\"Content-Type\":\"application/json\"}")
for name in ["API_KEY"]:
if not os.environ.get(name, "").strip():
raise RuntimeError("Set environment variable: " + name)
headers["Authorization"] = "Bearer " + os.environ["API_KEY"]
payload = json.loads("{\"model\":\"gemini-omni-1.2-flash\",\"prompt\":\"保持图片中的人物外观,让人物向镜头挥手。\",\"input_reference\":\"https://example.com/subject.png\",\"seconds\":\"8\",\"size\":\"720x1280\"}")
response = requests.request("POST", url, headers=headers, json=payload, timeout=120)
response.raise_for_status()
result = response.json()
if isinstance(result, dict) and (result.get("success") is False or result.get("code") is False or result.get("error")):
raise RuntimeError(json.dumps(result, ensure_ascii=False))
print(json.dumps(result, ensure_ascii=False, indent=2))
javascript
// Node.js 20+
const url = "https://api.tu-zi.com/v1/videos";
const headers = {
"Authorization": "Bearer YOUR_API_KEY",
"Content-Type": "application/json"
};
for (const name of ["API_KEY"]) {
if (!process.env[name]?.trim()) throw new Error("Set environment variable: " + name);
}
headers.Authorization = 'Bearer ' + process.env.API_KEY;
const response = await fetch(url, { method: "POST", headers, body: JSON.stringify({
"model": "gemini-omni-1.2-flash",
"prompt": "保持图片中的人物外观,让人物向镜头挥手。",
"input_reference": "https://example.com/subject.png",
"seconds": "8",
"size": "720x1280"
}), signal: AbortSignal.timeout(120000) });
if (!response.ok) throw new Error(await response.text());
const result = await response.json();
if (result?.success === false || result?.code === false || result?.error) throw new Error(JSON.stringify(result));
console.log(JSON.stringify(result, null, 2));
Multiple images and output format
Applies to: gemini-omni-1.1-flash, gemini-omni-1.2-flash
POST /v1/videos
{
"model": "gemini-omni-1.2-flash",
"input": [
{
"type": "text",
"text": "人物在第二张图的场景中行走。"
},
{
"type": "image",
"mime_type": "image/png",
"uri": "https://example.com/subject.png"
},
{
"type": "image",
"mime_type": "image/png",
"uri": "https://example.com/end.png"
}
],
"response_format": {
"type": "video",
"duration": "8s",
"resolution": "720p",
"aspect_ratio": "16:9",
"delivery": "uri"
}
}
curl
curl --fail-with-body --silent --show-error --connect-timeout 10 --max-time 120 --request POST 'https://api.tu-zi.com/v1/videos' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "gemini-omni-1.2-flash",
"input": [
{
"type": "text",
"text": "人物在第二张图的场景中行走。"
},
{
"type": "image",
"mime_type": "image/png",
"uri": "https://example.com/subject.png"
},
{
"type": "image",
"mime_type": "image/png",
"uri": "https://example.com/end.png"
}
],
"response_format": {
"type": "video",
"duration": "8s",
"resolution": "720p",
"aspect_ratio": "16:9",
"delivery": "uri"
}
}'
python
import json
import os
import requests
url = "https://api.tu-zi.com/v1/videos"
headers = json.loads("{\"Authorization\":\"Bearer YOUR_API_KEY\",\"Content-Type\":\"application/json\"}")
for name in ["API_KEY"]:
if not os.environ.get(name, "").strip():
raise RuntimeError("Set environment variable: " + name)
headers["Authorization"] = "Bearer " + os.environ["API_KEY"]
payload = json.loads("{\"model\":\"gemini-omni-1.2-flash\",\"input\":[{\"type\":\"text\",\"text\":\"人物在第二张图的场景中行走。\"},{\"type\":\"image\",\"mime_type\":\"image/png\",\"uri\":\"https://example.com/subject.png\"},{\"type\":\"image\",\"mime_type\":\"image/png\",\"uri\":\"https://example.com/end.png\"}],\"response_format\":{\"type\":\"video\",\"duration\":\"8s\",\"resolution\":\"720p\",\"aspect_ratio\":\"16:9\",\"delivery\":\"uri\"}}")
response = requests.request("POST", url, headers=headers, json=payload, timeout=120)
response.raise_for_status()
result = response.json()
if isinstance(result, dict) and (result.get("success") is False or result.get("code") is False or result.get("error")):
raise RuntimeError(json.dumps(result, ensure_ascii=False))
print(json.dumps(result, ensure_ascii=False, indent=2))
javascript
// Node.js 20+
const url = "https://api.tu-zi.com/v1/videos";
const headers = {
"Authorization": "Bearer YOUR_API_KEY",
"Content-Type": "application/json"
};
for (const name of ["API_KEY"]) {
if (!process.env[name]?.trim()) throw new Error("Set environment variable: " + name);
}
headers.Authorization = 'Bearer ' + process.env.API_KEY;
const response = await fetch(url, { method: "POST", headers, body: JSON.stringify({
"model": "gemini-omni-1.2-flash",
"input": [
{
"type": "text",
"text": "人物在第二张图的场景中行走。"
},
{
"type": "image",
"mime_type": "image/png",
"uri": "https://example.com/subject.png"
},
{
"type": "image",
"mime_type": "image/png",
"uri": "https://example.com/end.png"
}
],
"response_format": {
"type": "video",
"duration": "8s",
"resolution": "720p",
"aspect_ratio": "16:9",
"delivery": "uri"
}
}), signal: AbortSignal.timeout(120000) });
if (!response.ok) throw new Error(await response.text());
const result = await response.json();
if (result?.success === false || result?.code === false || result?.error) throw new Error(JSON.stringify(result));
console.log(JSON.stringify(result, null, 2));
Base64 video reference
Applies to: gemini-omni-1.1-flash, gemini-omni-1.2-flash
POST /v1/videos
Encode MP4 as raw Base64 and replace the data placeholder; public video URLs are unsupported. On 2026-11-15, a small business-site 1.1 sample failed generation with HTTP 400, and a larger input failed task persistence. Version 1.2 is untested; reference-video generation is not accepted.
{
"model": "gemini-omni-1.2-flash",
"input": [
{
"type": "text",
"text": "参考片段中的动作,创作新的花园场景。"
},
{
"type": "video",
"mime_type": "video/mp4",
"data": "REPLACE_WITH_RAW_BASE64_MP4"
}
],
"seconds": "8",
"size": "1280x720"
}
Edit a previous task (conditional)
Applies to: gemini-omni-1.1-flash, gemini-omni-1.2-flash
POST /v1/videos
Requires a completed continuable site task. On 2026-11-15, business-site 1.1 Flash failed with HTTP 400 despite correct fields and a fresh successful source task. Version 1.2 is untested; editing is not accepted.
{
"model": "gemini-omni-1.2-flash",
"prompt": "保留人物动作,把天空改为夕阳。",
"previous_interaction_id": "task_REPLACE_WITH_SUCCESSFUL_VIDEO",
"seconds": "8",
"size": "1280x720",
"generation_config": {
"video_config": {
"task": "edit"
}
}
}
Extend a previous task (conditional)
Applies to: gemini-omni-1.1-flash, gemini-omni-1.2-flash
POST /v1/videos
Continuation is limited to 10 seconds and requires eligible task access; output is unverified.
{
"model": "gemini-omni-1.2-flash",
"prompt": "继续上一段镜头,人物走向远处。",
"previous_interaction_id": "task_REPLACE_WITH_SUCCESSFUL_VIDEO",
"seconds": "8",
"generation_config": {
"video_config": {
"task": "extend"
}
}
}
Task results and downloads
Save id or task_id from submission. Poll GET /v1/videos/{task_id}: queued / in_progress lead to completed or failed. Read video_url on completion (some responses also expose metadata.url). Resolve relative URLs against the API origin. The site /content endpoint uses your site key; external downloads must not receive that key. Download expiring results promptly. Resume GET after a timeout rather than automatically repeating POST.
Complete unified video Python / JavaScript polling and download examples
Troubleshooting
| Symptom | Check |
|---|---|
| Image validation / URI error | Ensure a video or task ID was not placed in an image field; check the URL returns media |
| Unsupported content type / field | Check the exact family and content/contents/input schema; voice_id cannot be an audio URL |
| 400 conflicting fields | Check prompt/input, frames/references, and seconds/response_format exclusivity |
| HTTP success but task failed | Retain the site task ID, time, model, domain and redacted request; submission 200 is not output success |
| 404 or inaccessible task | Check domain, key access and task type; do not use IDs from another site or the official API |
| Local file cannot be fetched | Use a supported upload flow or a reachable HTTPS media URL |
Sources and scope
Use cases
- Choose video models and media formats
- Diagnose request and task failures
API protocols
/v1/videos
FAQ
Does a task ID prove generation succeeded?
No. Save the site task ID, poll to a successful terminal state and verify the result. Inspect failures instead of automatically creating a new task.
Can I put a local image, video or WAV path in JSON?
No. URL fields require reachable media, not localhost, disk paths or browser-login links. Kling PCM WAV requires an enabled multipart voice endpoint (both sites returned 404 on 2026-11-15). Follow each family’s asset workflow for other media.
Can parameters and task IDs be reused across models?
Do not assume compatibility. Check the exact model, endpoint, casing, media role and exclusive fields. Editing, drafts and voices require accessible successful tasks of the correct kind. Official capabilities are not evidence of site acceptance.
Related guides
Official sources
- Gemini Omni Official
兔子API