mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(sdk): preserve Gemini media parts in message conversion instead of dropping the turn (#3882)
## Description
The TypeScript SDK's Gemini converters drop media. `geminiToOpenAI()`
only carries `text` and `functionResponse` parts, so for a user turn:
- an **`inlineData`** part (the normal way to send an image to Gemini)
is silently dropped; an **image-only** turn produces no message at all
and the turn is deleted;
- a **`fileData`** part (Files API / URI reference) is dropped the same
way.
`openAIToGemini()` drops `image_url` parts on the way back, so even a
preserved image could not have survived the return trip.
Verified through the public `toOpenAI` / `fromOpenAI` on `main`:
```text
image-only | OpenAI: [] | back to Gemini: []
text+image | OpenAI: [{"role":"user","content":"what is this?"}] | back to Gemini: [{"role":"user","parts":[{"text":"what is this?"}]}]
text+fileData | OpenAI: [{"role":"user","content":"summarise"}] | back to Gemini: [{"role":"user","parts":[{"text":"summarise"}]}]
```
This hits the universal `compress()` for Gemini-format input and the
Gemini `withHeadroom()` adapter. The adapter always forwards the
round-tripped messages (it does not check `result.compressed`), so with
the adapter **every** vision request loses its image, including when the
proxy is unreachable and the client falls back. This is the Gemini
counterpart of #3439 (Anthropic image blocks), fixed the same way.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **`sdk/typescript/src/utils/format.ts`**
- `geminiToOpenAI` user branch (forward): `inlineData { mimeType, data
}` → `{ type: "image_url", image_url: { url:
"data:<mimeType>;base64,<data>" } }`; `fileData { mimeType, fileUri }` →
`image_url` with the URI plus a private `mime_type` key carrying the
MIME type. A user turn containing media is emitted as one `content` part
array preserving part order; unconvertible media parts are skipped
without deleting the turn. **Text-only turns are unchanged** (still a
newline-joined string) and `functionResponse` handling is untouched.
- `openAIToGemini` user branch (reverse): a `data:` URI → `inlineData {
mimeType, data }` (lossless); any other URL → `fileData { mimeType,
fileUri }` using the carried `mime_type`, else a type inferred from a
common image/PDF extension, else `fileUri` only. Text-only array content
still collapses to a single text part as before.
- Two small helpers, `geminiMediaImageUrl` / `geminiMediaPart`,
mirroring the Anthropic ones from #3439.
- **`sdk/typescript/test/utils/format.test.ts`** — first unit tests for
the Gemini converters (there were none): forward (inline image with
text, image-only turn, `fileData` with its MIME type carried), reverse
(data URI → `inlineData`, carried type → complete `fileData`, extension
inference, `fileUri`-only fallback), exact round trips for
text+`inlineData` and text+`fileData` (`application/pdf`), plus two
backward-compat guards for the text-only paths.
- **`sdk/typescript/test/adapters/gemini.test.ts`** — end-to-end through
`withHeadroom`: the image reaches `/v1/compress` as an `image_url` part
and comes back to `generateContent` as `inlineData` on a successful
compression; an image-only turn survives when the proxy is unreachable.
**Why the MIME type is carried:** `@google/generative-ai` declares
`FileData { mimeType: string; fileUri: string }` with both fields
required, so the OpenAI-shaped part must carry the type for the round
trip to be exact. The proxy's content walkers append non-text parts
unchanged (`headroom/proxy/helpers.py`, see the issue #3552 note there),
so the private `mime_type` key survives `/v1/compress`; it is never sent
to an OpenAI endpoint by the SDK's own paths. A foreign `image_url` with
no known type falls back to `fileUri` only (the REST schema marks
`mimeType` optional), and a test documents that fallback. `inlineData`
round-trips byte-for-byte via the data URI.
## Testing
- [x] Unit tests pass (`npm test` — vitest, incl. typecheck)
- [x] Type checking passes (`npm run typecheck` — tsc --noEmit)
- [x] Build passes (`npm run build` — tsup)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cd sdk/typescript && npx vitest run test/utils/format.test.ts test/adapters/gemini.test.ts
Test Files 2 passed (2)
Tests 54 passed (54)
$ npm test # full suite
Test Files 19 passed | 5 skipped (24)
Tests 350 passed | 33 skipped (383)
Type Errors no errors
$ npm run typecheck # tsc --noEmit
(clean, no errors)
$ npm run build # tsup
CJS / ESM / DTS Build success
```
## Real Behavior Proof
- Environment: macOS (Apple silicon), Node v22.16.0, npm 10.9.2; `cd
sdk/typescript && npm ci`.
- Exact command / steps: (1) the block above — `node
--input-type=module` script calling the built `toOpenAI` / `fromOpenAI`
from `dist/index.js` on `main`; (2) the ten new tests run on `main`
before the change and on this branch after; then `npm test`, `npm run
typecheck`, `npm run build`.
- Observed result: On `main` the eight behavioural tests fail for the
expected reason — e.g. `expected 'what is this?' to deeply equal [ {
type: 'text', … }, … ]` (the image part is missing) and the adapter
fallback test receives `contents` without the `inlineData` part (`[]`
messages). With the change all 54 targeted tests pass, including the
exact text+`fileData` (`application/pdf`) round trip; full suite 350
passed | 33 skipped; typecheck and build clean.
- Not tested: a live proxy round trip and a live Gemini call. The
conversion is pure and covered by the unit tests; the proxy consumes
standard OpenAI-format content parts (the same `image_url` parts #3439
already sends for Anthropic input).
## Runtime Rollout Safety
- Rollout-managed feature(s): None. Client-side TypeScript SDK
conversion fix; no server/runtime feature flag or rollout channel
involved.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Only the buggy path — media parts
that were dropped are now preserved. Text-only and `functionResponse`
conversions are byte-identical to before.
- Kill switch / disable path: N/A — correctness fix, not a gated
feature.
- Unsafe override required: No.
- Qualification impact: None — additive media handling; existing tests
unchanged and green.
- Rollback path: Revert this PR; no persisted state, migrations, or
config to unwind.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- Out of scope, left as-is: media parts in **model** (assistant) turns
are still dropped by `geminiToOpenAI`, and non-text content inside a
`functionResponse` is JSON-stringified. Both are pre-existing and rarer;
happy to follow up if wanted.
- The Gemini adapter forwarding round-tripped messages even when
`result.compressed` is false is also pre-existing; with this fix that
round trip is lossless for text and inline media, so the behaviour is
now harmless rather than destructive.
- CHANGELOG left untouched — the `no-manual-changelog` CI guard owns it;
the Conventional Commit title drives the entry.
This commit is contained in:
@@ -398,6 +398,44 @@ export function openAIToVercel(messages: OpenAIMessage[]): any[] {
|
||||
// Google Gemini → OpenAI
|
||||
// ============================================================
|
||||
|
||||
// Gemini inlineData / fileData part -> OpenAI image_url object, or null if unconvertible.
|
||||
// inlineData travels as a data: URI (MIME type included). fileData keeps its MIME type on a
|
||||
// private `mime_type` key so the reverse conversion can rebuild the complete
|
||||
// { mimeType, fileUri } that @google/generative-ai requires; the proxy passes non-text
|
||||
// parts through unchanged, so the key survives /v1/compress.
|
||||
function geminiMediaImageUrl(part: any): { url: string; mime_type?: string } | null {
|
||||
const blob = part.inlineData;
|
||||
if (blob && typeof blob === "object" && blob.mimeType && blob.data) {
|
||||
return { url: `data:${blob.mimeType};base64,${blob.data}` };
|
||||
}
|
||||
const file = part.fileData;
|
||||
if (file && typeof file === "object" && typeof file.fileUri === "string") {
|
||||
return typeof file.mimeType === "string"
|
||||
? { url: file.fileUri, mime_type: file.mimeType }
|
||||
: { url: file.fileUri };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
const MIME_BY_EXTENSION: Record<string, string> = {
|
||||
png: "image/png",
|
||||
jpg: "image/jpeg",
|
||||
jpeg: "image/jpeg",
|
||||
gif: "image/gif",
|
||||
webp: "image/webp",
|
||||
pdf: "application/pdf",
|
||||
};
|
||||
|
||||
// OpenAI image_url -> Gemini part. data: URIs become inlineData. Other urls become fileData
|
||||
// with the carried MIME type, else one inferred from a common extension, else fileUri only.
|
||||
function geminiMediaPart(imageUrl: { url: string; mime_type?: string }): any {
|
||||
const m = /^data:([^;,]+);base64,(.*)$/s.exec(imageUrl.url);
|
||||
if (m) return { inlineData: { mimeType: m[1], data: m[2] } };
|
||||
const ext = /\.([a-z0-9]+)(?:[?#]|$)/i.exec(imageUrl.url)?.[1]?.toLowerCase();
|
||||
const mimeType = imageUrl.mime_type ?? (ext ? MIME_BY_EXTENSION[ext] : undefined);
|
||||
return { fileData: mimeType ? { mimeType, fileUri: imageUrl.url } : { fileUri: imageUrl.url } };
|
||||
}
|
||||
|
||||
export function geminiToOpenAI(messages: any[]): OpenAIMessage[] {
|
||||
const result: OpenAIMessage[] = [];
|
||||
|
||||
@@ -408,10 +446,26 @@ export function geminiToOpenAI(messages: any[]): OpenAIMessage[] {
|
||||
if (role === "user") {
|
||||
// Check for functionResponse parts
|
||||
const funcResponses = parts.filter((p: any) => p.functionResponse);
|
||||
const textParts = parts.filter((p: any) => p.text !== undefined);
|
||||
const hasMedia = parts.some((p: any) => p.inlineData || p.fileData);
|
||||
|
||||
if (textParts.length > 0) {
|
||||
result.push({ role: "user", content: textParts.map((p: any) => p.text).join("\n") });
|
||||
if (hasMedia) {
|
||||
const content = parts
|
||||
.filter((p: any) => p.text !== undefined || p.inlineData || p.fileData)
|
||||
.map((p: any) => {
|
||||
if (p.text !== undefined) return { type: "text" as const, text: p.text };
|
||||
const imageUrl = geminiMediaImageUrl(p);
|
||||
return imageUrl ? { type: "image_url" as const, image_url: imageUrl } : null;
|
||||
})
|
||||
.filter((p: any): p is NonNullable<typeof p> => p !== null);
|
||||
|
||||
if (content.length > 0) {
|
||||
result.push({ role: "user", content });
|
||||
}
|
||||
} else {
|
||||
const textParts = parts.filter((p: any) => p.text !== undefined);
|
||||
if (textParts.length > 0) {
|
||||
result.push({ role: "user", content: textParts.map((p: any) => p.text).join("\n") });
|
||||
}
|
||||
}
|
||||
for (const fr of funcResponses) {
|
||||
result.push({
|
||||
@@ -459,6 +513,17 @@ export function openAIToGemini(messages: OpenAIMessage[]): any[] {
|
||||
}
|
||||
|
||||
if (msg.role === "user") {
|
||||
if (Array.isArray(msg.content) && msg.content.some((p) => p.type === "image_url")) {
|
||||
const geminiParts = msg.content
|
||||
.map((p) => {
|
||||
if (p.type === "text") return { text: p.text };
|
||||
if (p.type === "image_url") return geminiMediaPart(p.image_url);
|
||||
return null;
|
||||
})
|
||||
.filter((p) => p !== null);
|
||||
result.push({ role: "user", parts: geminiParts });
|
||||
continue;
|
||||
}
|
||||
const text = typeof msg.content === "string"
|
||||
? msg.content
|
||||
: (msg.content ?? []).filter((p) => p.type === "text").map((p) => (p as any).text).join("\n");
|
||||
|
||||
@@ -128,4 +128,70 @@ describe("Gemini withHeadroom", () => {
|
||||
|
||||
expect(result).toBe(originalResult);
|
||||
});
|
||||
|
||||
it("preserves inline image parts through compression", async () => {
|
||||
const mockModel = {
|
||||
generateContent: vi.fn().mockResolvedValue({ response: { text: () => "a cat" } }),
|
||||
};
|
||||
// The proxy keeps the image part (it only compresses text).
|
||||
mockFetch.mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "what is this?" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,AAAA" } },
|
||||
],
|
||||
},
|
||||
],
|
||||
tokens_before: 500,
|
||||
tokens_after: 100,
|
||||
tokens_saved: 400,
|
||||
compression_ratio: 0.2,
|
||||
transforms_applied: ["smart_crusher"],
|
||||
ccr_hashes: [],
|
||||
}),
|
||||
});
|
||||
|
||||
const wrapped = withHeadroom(mockModel, { baseUrl: "http://test:8787" });
|
||||
await wrapped.generateContent({
|
||||
contents: [
|
||||
{
|
||||
role: "user",
|
||||
parts: [{ text: "what is this?" }, { inlineData: { mimeType: "image/png", data: "AAAA" } }],
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
// The image must reach the proxy as an image_url part ...
|
||||
const body = JSON.parse(mockFetch.mock.calls[0][1].body);
|
||||
expect(body.messages[0].content).toEqual([
|
||||
{ type: "text", text: "what is this?" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,AAAA" } },
|
||||
]);
|
||||
// ... and come back to the model as an inlineData part.
|
||||
expect(mockModel.generateContent).toHaveBeenCalledWith({
|
||||
contents: [
|
||||
{
|
||||
role: "user",
|
||||
parts: [{ text: "what is this?" }, { inlineData: { mimeType: "image/png", data: "AAAA" } }],
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps an image-only turn when the proxy is unreachable", async () => {
|
||||
const mockModel = { generateContent: vi.fn().mockResolvedValue({}) };
|
||||
mockFetch.mockRejectedValue(new Error("connect ECONNREFUSED"));
|
||||
|
||||
const wrapped = withHeadroom(mockModel, { baseUrl: "http://test:8787" });
|
||||
const contents = [
|
||||
{ role: "user", parts: [{ inlineData: { mimeType: "image/png", data: "AAAA" } }] },
|
||||
];
|
||||
await wrapped.generateContent({ contents });
|
||||
|
||||
expect(mockModel.generateContent).toHaveBeenCalledWith({ contents });
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,5 +1,12 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { vercelToOpenAI, openAIToVercel, anthropicToOpenAI, openAIToAnthropic } from "../../src/utils/format.js";
|
||||
import {
|
||||
vercelToOpenAI,
|
||||
openAIToVercel,
|
||||
anthropicToOpenAI,
|
||||
openAIToAnthropic,
|
||||
geminiToOpenAI,
|
||||
openAIToGemini,
|
||||
} from "../../src/utils/format.js";
|
||||
import type { OpenAIMessage } from "../../src/types.js";
|
||||
|
||||
describe("vercelToOpenAI", () => {
|
||||
@@ -639,6 +646,158 @@ describe("round-trip conversion", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("geminiToOpenAI", () => {
|
||||
it("converts an inlineData part to an image_url data URI and keeps the text", () => {
|
||||
const result = geminiToOpenAI([
|
||||
{
|
||||
role: "user",
|
||||
parts: [
|
||||
{ text: "what is this?" },
|
||||
{ inlineData: { mimeType: "image/png", data: "AAAA" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
expect(result).toEqual([
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "what is this?" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,AAAA" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps an image-only user turn instead of dropping it", () => {
|
||||
const result = geminiToOpenAI([
|
||||
{ role: "user", parts: [{ inlineData: { mimeType: "image/jpeg", data: "BBBB" } }] },
|
||||
]);
|
||||
expect(result).toEqual([
|
||||
{
|
||||
role: "user",
|
||||
content: [{ type: "image_url", image_url: { url: "data:image/jpeg;base64,BBBB" } }],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("converts a fileData part to an image_url carrying its fileUri", () => {
|
||||
const result = geminiToOpenAI([
|
||||
{
|
||||
role: "user",
|
||||
parts: [
|
||||
{ text: "summarise" },
|
||||
{ fileData: { mimeType: "application/pdf", fileUri: "https://generativelanguage.googleapis.com/v1beta/files/abc" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
expect(result).toEqual([
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "summarise" },
|
||||
{
|
||||
type: "image_url",
|
||||
image_url: { url: "https://generativelanguage.googleapis.com/v1beta/files/abc", mime_type: "application/pdf" },
|
||||
},
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("leaves text-only user turns as a newline-joined string (backward compat)", () => {
|
||||
const result = geminiToOpenAI([
|
||||
{ role: "user", parts: [{ text: "a" }, { text: "b" }] },
|
||||
]);
|
||||
expect(result).toEqual([{ role: "user", content: "a\nb" }]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("openAIToGemini", () => {
|
||||
it("restores an image_url data URI as an inlineData part", () => {
|
||||
const msgs: OpenAIMessage[] = [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "hi" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,AAAA" } },
|
||||
],
|
||||
},
|
||||
];
|
||||
expect(openAIToGemini(msgs)).toEqual([
|
||||
{ role: "user", parts: [{ text: "hi" }, { inlineData: { mimeType: "image/png", data: "AAAA" } }] },
|
||||
]);
|
||||
});
|
||||
|
||||
it("restores a non-data image_url with a carried MIME type as a complete fileData part", () => {
|
||||
const msgs = [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{
|
||||
type: "image_url",
|
||||
image_url: { url: "https://generativelanguage.googleapis.com/v1beta/files/abc", mime_type: "application/pdf" },
|
||||
},
|
||||
],
|
||||
},
|
||||
] as OpenAIMessage[];
|
||||
expect(openAIToGemini(msgs)).toEqual([
|
||||
{
|
||||
role: "user",
|
||||
parts: [{ fileData: { mimeType: "application/pdf", fileUri: "https://generativelanguage.googleapis.com/v1beta/files/abc" } }],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("infers the MIME type from a common file extension when the image_url carries none", () => {
|
||||
const msgs: OpenAIMessage[] = [
|
||||
{ role: "user", content: [{ type: "image_url", image_url: { url: "https://example.com/cat.png?size=large" } }] },
|
||||
];
|
||||
expect(openAIToGemini(msgs)).toEqual([
|
||||
{ role: "user", parts: [{ fileData: { mimeType: "image/png", fileUri: "https://example.com/cat.png?size=large" } }] },
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps only fileUri when no MIME type is known (documented fallback)", () => {
|
||||
const msgs: OpenAIMessage[] = [
|
||||
{ role: "user", content: [{ type: "image_url", image_url: { url: "https://example.com/blob" } }] },
|
||||
];
|
||||
expect(openAIToGemini(msgs)).toEqual([
|
||||
{ role: "user", parts: [{ fileData: { fileUri: "https://example.com/blob" } }] },
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps text-only array content as a single text part (backward compat)", () => {
|
||||
const msgs: OpenAIMessage[] = [
|
||||
{ role: "user", content: [{ type: "text", text: "a" }, { type: "text", text: "b" }] },
|
||||
];
|
||||
expect(openAIToGemini(msgs)).toEqual([{ role: "user", parts: [{ text: "a\nb" }] }]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("round-trip: geminiToOpenAI then openAIToGemini", () => {
|
||||
it("reproduces a text+inlineData turn exactly", () => {
|
||||
const original = [
|
||||
{
|
||||
role: "user",
|
||||
parts: [{ text: "what is this?" }, { inlineData: { mimeType: "image/png", data: "AAAA" } }],
|
||||
},
|
||||
];
|
||||
expect(openAIToGemini(geminiToOpenAI(original))).toEqual(original);
|
||||
});
|
||||
|
||||
it("reproduces a text+fileData (application/pdf) turn exactly", () => {
|
||||
const original = [
|
||||
{
|
||||
role: "user",
|
||||
parts: [
|
||||
{ text: "summarise" },
|
||||
{ fileData: { mimeType: "application/pdf", fileUri: "https://generativelanguage.googleapis.com/v1beta/files/abc" } },
|
||||
],
|
||||
},
|
||||
];
|
||||
expect(openAIToGemini(geminiToOpenAI(original))).toEqual(original);
|
||||
});
|
||||
});
|
||||
describe("openAIToVercel tool names", () => {
|
||||
it("carries the tool name from the matching tool-call onto the tool-result part", () => {
|
||||
const msgs: OpenAIMessage[] = [
|
||||
|
||||
Reference in New Issue
Block a user