mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(sdk): preserve Vercel AI SDK file and image parts through compression (#4048)
## Description
The TypeScript SDK's Vercel AI SDK converters lose or break media:
- **`headroomMiddleware()` / `withHeadroom(model)` drop every image and
document.** The provider prompt the middleware receives
(`params.prompt`) carries media only as `file` parts — in the installed
`@ai-sdk/provider` 4.0.19 a user message is
`Array<LanguageModelV3TextPart | LanguageModelV3FilePart>`; there is no
`image` part at that level. `vercelToOpenAI()` only handles `text` and
`image`, so a `file` part is silently removed and a file-only turn
becomes `content: []`. Because the middleware returns the original
params when compression did not happen, the media is lost exactly when
compression succeeds.
- **`compressVercelMessages()` crashes on binary images.** For
user-level `ModelMessage`s, an `image` part whose data is a `Uint8Array`
(or a bare base64 string) is stringified on the way out
(`"137,80,78,71"`) and `openAIToVercel()` then calls `new URL()` on it:
`TypeError: Invalid URL`, after a successful compression. `file` parts
(e.g. a PDF) are dropped on this path too.
Reproduced through the public `toOpenAI` / `fromOpenAI` on `main`
(`20ce054f`):
```text
prompt: text+file(image/png, bytes) | OpenAI: {"role":"user","content":"what is this?"} | back: text only
prompt: file-only (application/pdf) | OpenAI: {"role":"user","content":[]} | back: {"role":"user","content":[]}
message: image as Uint8Array | OpenAI: image_url.url = "137,80,78,71" | ROUND TRIP THROWS TypeError: Invalid URL
message: image as base64 string | OpenAI: image_url.url = "iVBORw0KGgo=" | ROUND TRIP THROWS TypeError: Invalid URL
message: file part (pdf) | OpenAI: {"role":"user","content":"summarise"} | back: text only
```
Seven more, found in review (JerrettDavis and the Devin bot,
2026-10-09/14) and fixed here as well:
- **The public `compress()` could not even detect a media-only Vercel
conversation.** `detectFormat` knew Vercel only by `tool-call` /
`tool-result` parts, so a conversation whose only Vercel markers were
`file` / `image` parts was classified as OpenAI and sent as-is — a
`Uint8Array` serialises as `{"0":137,"1":80,…}` on the wire.
- **Assistant file parts were dropped by the explicit converters too.**
AI SDK 6's `AssistantContent` includes `FilePart` (generated media sent
back as history) and this package's peer range is `ai >=6.0.0`, but the
assistant branch of `vercelToOpenAI` kept only text and tool calls.
- **Assistant media was dropped again when converting back to Anthropic
or Gemini.** The first revision's `assistantText` helper kept only the
text of an assistant content array, so an assistant image or file in the
OpenAI shape (e.g. from a Vercel file part) vanished in `fromOpenAI(…,
"anthropic" | "gemini")` even though both targets have media parts.
- **AI SDK 7 / LanguageModelV4 file parts were dropped by the
middleware.** `wrapLanguageModel` in `ai` 7.x hands `transformParams` a
V4 prompt whose `file` parts carry `data` as a tagged union (`{ type:
"data" | "url" | "text" | "reference", … }`), not the bare
bytes/string/URL the converter read, so the attachment vanished before
compression. Provider references were silently removed too.
- **A conversation whose only Anthropic marker is a `document` block was
detected as OpenAI**, so `compress()` skipped the Anthropic converter —
and once detected, the Anthropic user branch dropped the document, since
it only knew images.
- **Adjacent assistant text parts lost their newline join** in the
Anthropic and Gemini reverse converters.
- **Anthropic document sources other than PDF were dropped, and document
fields were lost.** A document with a plain-text source (or a content /
file source) produced no media part — a document-only turn vanished —
and a PDF document came back without its `title`, `context` and
`citations`.
Same bug class as #3439 (Anthropic) and #3882 (Gemini), on the Vercel
adapter, fixed the same way: media becomes OpenAI `image_url` parts the
proxy passes through untouched, with the few facts the OpenAI shape
cannot hold carried on private keys so the reverse conversion rebuilds
the right Vercel part.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **`sdk/typescript/src/utils/format.ts`**
- `vercelToOpenAI` user branch (forward): `image` **and** `file` parts
are converted. `Uint8Array` / `ArrayBuffer` / base64-string data →
`image_url` with a `data:<mediaType>;base64,…` URI (the media type
travels inside it; if the part has none, an empty media type is carried
rather than guessed); `URL` objects and URL strings → the URL, with the
part's media type on a private `mime_type` key (the same private key
#3882 introduced for Gemini, so one key serves every converter). A
`file` part additionally carries `part: "file"` and its `filename`.
Unconvertible data (e.g. provider file references) is skipped without
deleting the turn. A turn with media is emitted as an ordered part
array; **text-only turns are unchanged** (still joined into a string).
- **LanguageModelV4 / AI SDK 7 tagged data:** the carrier records the
tag on a private `data_tag` key — `data` and `text` travel as data: URIs
and are restored tagged (base64 inside `data`, decoded text inside
`text`), `url` is restored as a `URL` object inside the tag, and a
`reference` (tagged, or a bare AI SDK 6 `ProviderReference`) is carried
verbatim on a `reference` key with an empty data: URI as the opaque url
and returned untouched instead of being removed. Bare AI SDK 6 inputs
restore bare, so standalone `ModelMessage` handling is unchanged.
- `openAIToVercel` user branch (reverse): a data URI → base64 data plus
its media type; an http(s) URL → a `URL` object (unchanged behaviour,
existing test kept); anything else → left as a string. **Never throws.**
Emits `{ type: "file", mediaType, data, filename? }` when `part ===
"file"` or the media type is not `image/*`, else `{ type: "image",
image, mediaType? }`. Base64 strings are valid data for both the
provider prompt (`LanguageModelV3DataContent = Uint8Array | string |
URL`, string = base64) and `ModelMessage` (`DataContent` accepts
base64), so byte inputs come back as base64 strings — equal content, not
the same object.
- `detectFormat`: a `file` part carrying `data` / `mediaType` (OpenAI's
`file` part carries a `file` object instead) or an `image` part carrying
`image` (Anthropic's carries `source`) now marks the Vercel format; an
Anthropic `document` block with a `source.type` marks Anthropic. OpenAI
`image_url` parts and Anthropic `image` blocks are unaffected, with
tests for both.
- `vercelToOpenAI` assistant branch: a turn with `file` parts becomes an
ordered part array (text + `image_url` with the carriers above);
text-only assistant turns still convert to a string. `openAIToVercel`
rebuilds text and `file` parts with the assistant role.
- `openAIToAnthropic` / `openAIToGemini` assistant branches: a text-only
array joins with `"\n"` into one block/part, as the forward converters
join text-only turns; an array carrying media keeps every part separate,
because the forward converters keep exact parts in that case and a round
trip must not add or move characters (`groupAssistantParts`); assistant
media in the OpenAI shape is rebuilt in order relative to text —
Anthropic `image/*` → image block, `application/pdf` → document block
(base64 or url source, per the SDK's `DocumentBlockParam`); Gemini data
URIs → `inlineData`, URL-backed files → `fileData` with the carried MIME
type. Media Anthropic has no block for is omitted rather than mis-typed.
The Anthropic user branch now maps PDFs to document blocks as well.
- `anthropicToOpenAI` user branch: `document` blocks are kept beside
images — base64 PDF → data: URI; url source → url with `mime_type`;
plain-text source → `data:text/plain;base64,…` tagged `source_type:
"text"` and restored as text; content and file sources carried verbatim
on a `source` key (preserved, not removed); `title` / `context` /
`citations` / `cache_control` carried on a `document` key and
re-attached — so every `DocumentBlockParam` round-trips exactly and a
document-only turn is no longer dropped.
- `anthropicToOpenAI` assistant and `geminiToOpenAI` model branches:
image/document blocks and `inlineData`/`fileData` become ordered parts,
so both formats round-trip exactly (Gemini image-generation output is a
real case).
- Helpers: `bytesToBase64` / `base64ToText` (use `Buffer` when present,
`btoa`/`atob` otherwise), `parseUrl`, `isProviderReference`,
`vercelMediaImageUrl`, `vercelMediaPart`, `groupAssistantParts`,
`anthropicMediaImageUrl` / `anthropicMediaBlock` (with the
`AnthropicMediaImageUrl` carrier and `ANTHROPIC_DOCUMENT_FIELDS`).
- **`sdk/typescript/src/types.ts`** — `AssistantMessage.content` widens
from `string | null` to `string | ContentPart[] | null` so an assistant
turn can carry media parts; OpenAI's own API already accepts an array of
parts as assistant content. Additive for producers; the only readers in
the SDK (`hooks.ts`) already branch on arrays, and the
`plugins/openclaw` converter defines its own message type. The public
type test is updated to match.
- **`sdk/typescript/test/utils/detect-format.test.ts`** — media-only
Vercel turns (file part; image part) detect as `vercel`; a document-only
Anthropic turn detects as `anthropic`; OpenAI `image_url` and Anthropic
`image` detection unchanged.
- **`sdk/typescript/test/compress.test.ts`** — the public `compress()`
with a media-only Vercel turn and a mocked proxy: the wire body carries
an `image_url` data URI (and no `"0":137`), and the result comes back as
a Vercel `file` part; a document-only Anthropic turn through
`compress()` against a pass-through mock comes back as the same document
block.
- **`sdk/typescript/test/utils/format.test.ts`** — Anthropic document
sources (document-only text turn through detection and conversion, mixed
text+document with context, content and file sources verbatim, PDF
fields, assistant document); V4 tagged data (data, url, text, reference,
and a bare reference) forward/reverse/round trip; assistant text
separators (text-only join; media-bearing arrays keep parts; exact
Gemini and Anthropic round trips with adjacent text before media);
Anthropic user documents (text+document, document-only, round trip
through detection); cross-format assistant media: the review thread's
Vercel → OpenAI → Gemini / Anthropic example, both reverse converters
(image; PDF base64 and url; text-only backward compat), both forward
converters, exact Anthropic and Gemini assistant round trips; Vercel
assistant: text + file forward, reverse with the assistant role, round
trip, text-only backward compat; user: forward (prompt `file` image
bytes with text; file-only PDF with filename; `Uint8Array` image;
bare-base64 image; URL-backed file with carried media type), reverse
(data URI marked `file`; image data URI; URL-backed file; unparseable
url does not throw), and exact round trips (prompt text+file; byte
image).
- **`sdk/typescript/test/adapters/vercel-ai.test.ts`** —
`headroomMiddleware`: a prompt `file` part reaches `/v1/compress` as an
`image_url` data URI and comes back in the returned prompt as a `file`
part; the same for an assistant file turn in the prompt; a public
`transformParams` round trip typed with the actual
`LanguageModelV4FilePart` / `LanguageModelV4Prompt` (tagged data
restored tagged) and a provider reference passed through untouched;
`compressVercelMessages` no longer throws on byte image data.
## Testing
- [x] Unit tests pass (`npm test` — vitest, incl. typecheck; also run by
the new `sdk-typescript.yml` job)
- [x] Type checking passes (`npm run typecheck` — tsc --noEmit)
- [x] Build passes (`npm run build` — tsup)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cd sdk/typescript && npx vitest run test/utils/format.test.ts test/utils/detect-format.test.ts test/compress.test.ts \
test/adapters/vercel-ai.test.ts test/adapters/anthropic.test.ts test/adapters/gemini.test.ts
Test Files 6 passed (6)
Tests 147 passed (147)
$ npm test # full suite
Test Files 19 passed | 5 skipped (24)
Tests 404 passed | 33 skipped (437)
Type Errors no errors
$ npm run typecheck # tsc --noEmit
(clean, no errors)
$ npm run build # tsup
CJS / ESM / DTS Build success
```
## Real Behavior Proof
- Environment: macOS (Apple silicon), Node v22.16.0, npm 10.9.2; `cd
sdk/typescript && npm ci` (`ai` 7.0.119, `@ai-sdk/provider` 4.0.19,
`@ai-sdk/provider-utils` 5.0.50).
- Exact command / steps: (1) the block in the description — a `node
--input-type=module` script calling the built `toOpenAI` / `fromOpenAI`
from `dist/index.js` on `main`; (2) the thirteen new tests run on `main`
before the change and on this branch after; (3) for the review findings:
`detectFormat` + `toOpenAI` on a media-only Vercel turn, and the
converters on an assistant text+file turn, both against the first
revision; then the eight new tests before and after the second commit;
(4) second review round: `fromOpenAI(vercelToOpenAI(<assistant text +
file>), "gemini")` on the second revision, then the eleven new tests
before and after the third commit; then `npm test`, `npm run typecheck`,
`npm run build`.
- Observed result: on `main` all thirteen fail for the expected reasons
— media parts missing from the converted output, and `AssertionError:
expected [Function] to not throw an error but 'TypeError: Invalid URL'
was thrown` for the byte/base64 cases (the middleware test fails the
same way). With the first commit all 71 targeted tests pass. Review
findings reproduced on that revision: `detect = openai` and the wire
payload
`{"type":"file","mediaType":"image/png","data":{"0":137,"1":80,"2":78,"3":71}}`
for a media-only turn; `{role:"assistant",content:"generated image"}`
for an assistant text+file turn (file gone). The eight new tests fail on
that revision (seven runtime, one type assertion) and pass after the
second commit. Second round: on that revision the Gemini conversion of a
Vercel assistant file returned `{ role: "model", parts: [{ text:
"generated image" }] }` (media gone); the eleven new tests fail there
and pass after the third commit: 127 targeted tests, full suite 384
passed | 33 skipped, typecheck and build clean.
- **Live proxy round trip** (replaces the earlier "not tested"):
`headroom proxy` 0.40.0 from the published wheel (`uvx --from
'headroom-ai[proxy]' headroom proxy --port 8799`), the built middleware
from this branch, a V4 prompt with a 120-line service log, a data-tagged
PNG, a provider-reference PDF and an assistant file turn; then the
public `compress()` with a media-only Vercel turn. Captured on
`b00b388b`; the later `f5dbdbcc` changes only the Anthropic user branch
and the text-part rule, not the Vercel path exercised here. Output (text
parts shown as `text(<length>)`):
```text
proxy http://127.0.0.1:8799/v1/compress status 200 | tokens_before 4497 tokens_after 532 tokens_saved 3965 | transforms ["router:text_block:log"]
SENT : [["text(10758)","{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64,iVBORw==\",\"part\":\"file\",\"data_tag\":\"data\"}}","{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:application/pdf;base64,\",\"part\":\"file\",\"data_tag\":\"reference\",\"reference\":{\"openai\":\"file-abc\"}}}"],["text(15)","{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64,iVBORw==\",\"part\":\"file\",\"data_tag\":\"data\"}}"]]
RETURNED : [["text(29)","{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64,iVBORw==\",\"part\":\"file\",\"data_tag\":\"data\"}}","{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:application/pdf;base64,\",\"part\":\"file\",\"data_tag\":\"reference\",\"reference\":{\"openai\":\"file-abc\"}}}"],["text(15)","{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64,iVBORw==\",\"part\":\"file\",\"data_tag\":\"data\"}}"]]
RESTORED : [["text(29)","{\"type\":\"file\",\"mediaType\":\"image/png\",\"data\":{\"type\":\"data\",\"data\":\"iVBORw==\"}}","{\"type\":\"file\",\"mediaType\":\"application/pdf\",\"data\":{\"type\":\"reference\",\"reference\":{\"openai\":\"file-abc\"}}}"],["text(15)","{\"type\":\"file\",\"mediaType\":\"image/png\",\"data\":{\"type\":\"data\",\"data\":\"iVBORw==\"}}"]]
middleware returned original params (no compression)? false
compress(): sent part[1] = {"type":"image_url","image_url":{"url":"data:image/png;base64,iVBORw==","part":"file"}} | returned part[1] = {"type":"image_url","image_url":{"url":"data:image/png;base64,iVBORw==","part":"file"}} | result part[1] = {"type":"file","mediaType":"image/png","data":"iVBORw=="} | compressed = true saved 3950
```
The proxy compressed the log, returned every media part with the private
carrier keys intact, and the restored prompt carries the tagged V4 parts
and the reference unchanged; the public `compress()` path detected
Vercel, compressed, and restored the file part.
- Not tested: a live provider call downstream of the proxy.
## Runtime Rollout Safety
- Rollout-managed feature(s): None. Client-side TypeScript SDK
conversion fix; no server/runtime feature flag or rollout channel
involved.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Only the buggy paths — media that was
dropped is preserved, inputs that threw now convert, and media-only
Vercel conversations are now detected as Vercel. Text-only turns, tool
calls/results and the http-URL image case are byte-identical to before.
The one public type change is additive: `AssistantMessage.content` also
admits an array of content parts.
- Kill switch / disable path: N/A — correctness fix, not a gated
feature.
- Unsafe override required: No.
- Qualification impact: None — additive media handling; existing
conversion tests unchanged and green.
- Rollback path: Revert this PR; no persisted state, migrations, or
config to unwind.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- Out of scope, left as-is and pre-existing: assistant `reasoning` parts
(dropped by the assistant branch of `vercelToOpenAI`); media inside
`tool-result` parts; `providerOptions` on parts. Each is a separate
follow-up.
- Anthropic has blocks only for images and PDFs (and text documents);
other media reaching `openAIToAnthropic` is omitted rather than emitted
as a mis-typed block.
- Revisions after review: detection of media-only Vercel conversations
and assistant file parts (second commit), including the
`AssistantMessage.content` widening described above; assistant media
through the Anthropic and Gemini converters in both directions, the
`mime_type` carrier rename and the removal of `assistantText` (third
commit); LanguageModelV4 tagged data and provider references, Anthropic
document detection, assistant text separators and the live proxy proof
(fourth commit); user documents kept in the Anthropic user branch and
exact text parts beside media (fifth commit); Anthropic text / content /
file document sources and document fields (sixth commit, on top of the
maintainer's `main` sync).
- Large documents travel as base64 inside the request body, as
`inlineData` already does for Gemini (#3882); the proxy counts image
parts at a fixed cost.
- CHANGELOG left untouched — the `no-manual-changelog` CI guard owns it;
the Conventional Commit title drives the entry.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
125a703a65
commit
1668a6c0dd
@@ -38,7 +38,11 @@ export interface UserMessage {
|
||||
|
||||
export interface AssistantMessage {
|
||||
role: "assistant";
|
||||
content: string | null;
|
||||
/**
|
||||
* A string, or an ordered part array when the assistant turn carries media
|
||||
* (e.g. a Vercel AI SDK file part): text parts plus `image_url` parts.
|
||||
*/
|
||||
content: string | ContentPart[] | null;
|
||||
tool_calls?: ToolCall[];
|
||||
}
|
||||
|
||||
|
||||
@@ -14,6 +14,7 @@
|
||||
import type {
|
||||
OpenAIMessage,
|
||||
AssistantMessage,
|
||||
ContentPart,
|
||||
ToolCall,
|
||||
} from "../types.js";
|
||||
|
||||
@@ -51,8 +52,14 @@ export function detectFormat(messages: any[]): MessageFormat {
|
||||
if (part.type === "tool-call" || part.type === "tool-result") return "vercel";
|
||||
// Anthropic: underscored types
|
||||
if (part.type === "tool_use" || part.type === "tool_result") return "anthropic";
|
||||
// Anthropic: image with source.type
|
||||
if (part.type === "image" && part.source?.type) return "anthropic";
|
||||
// Anthropic: image / document with source.type
|
||||
if ((part.type === "image" || part.type === "document") && part.source?.type) return "anthropic";
|
||||
// Vercel: `file` parts carry data/mediaType (OpenAI's `file` part carries a `file` object);
|
||||
// `image` parts carry `image` (Anthropic's carry `source`)
|
||||
if (part && typeof part === "object") {
|
||||
if (part.type === "file" && ("data" in part || "mediaType" in part)) return "vercel";
|
||||
if (part.type === "image" && "image" in part) return "vercel";
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -88,6 +95,74 @@ function openAIImageBlock(url: string): any {
|
||||
return { type: "image", source: { type: "url", url } };
|
||||
}
|
||||
|
||||
type AnthropicMediaImageUrl = {
|
||||
url: string;
|
||||
mime_type?: string;
|
||||
// A plain-text document source travels as a data: URI and is tagged so it is restored as
|
||||
// text, not base64; sources the OpenAI shape cannot express (content, file) are carried
|
||||
// verbatim on `source`. The document's own fields are carried on `document`.
|
||||
source_type?: "text";
|
||||
source?: any;
|
||||
document?: Record<string, any>;
|
||||
};
|
||||
|
||||
const ANTHROPIC_DOCUMENT_FIELDS = ["title", "context", "citations", "cache_control"] as const;
|
||||
|
||||
// Anthropic media block -> OpenAI image_url object, or null if unconvertible: image blocks via
|
||||
// anthropicImageUrl; document blocks by source — base64 PDF as a data: URI, url with the MIME
|
||||
// type carried on the private `mime_type` key, plain text as a tagged data: URI, and content /
|
||||
// file sources carried verbatim — plus the document's title / context / citations / cache_control.
|
||||
function anthropicMediaImageUrl(block: any): AnthropicMediaImageUrl | null {
|
||||
if (block.type === "image") {
|
||||
const url = anthropicImageUrl(block.source);
|
||||
return url ? { url } : null;
|
||||
}
|
||||
if (block.type !== "document" || !block.source || typeof block.source !== "object") return null;
|
||||
const src = block.source;
|
||||
let out: AnthropicMediaImageUrl;
|
||||
if (src.type === "base64" && src.media_type && src.data) {
|
||||
out = { url: `data:${src.media_type};base64,${src.data}` };
|
||||
} else if (src.type === "url" && typeof src.url === "string") {
|
||||
out = { url: src.url, mime_type: "application/pdf" };
|
||||
} else if (src.type === "text" && typeof src.data === "string") {
|
||||
out = {
|
||||
url: `data:${src.media_type || "text/plain"};base64,${bytesToBase64(new TextEncoder().encode(src.data))}`,
|
||||
source_type: "text",
|
||||
};
|
||||
} else {
|
||||
out = { url: "data:;base64,", source: src };
|
||||
}
|
||||
const fields: Record<string, any> = {};
|
||||
for (const k of ANTHROPIC_DOCUMENT_FIELDS) if (block[k] !== undefined) fields[k] = block[k];
|
||||
if (Object.keys(fields).length > 0) out.document = fields;
|
||||
return out;
|
||||
}
|
||||
|
||||
// OpenAI image_url -> Anthropic media block, or null when Anthropic has no block for that media:
|
||||
// a carried document source or text document is rebuilt as a document block with its fields;
|
||||
// application/pdf -> document block (base64 or url source); image/* (or unknown) -> image block.
|
||||
function anthropicMediaBlock(imageUrl: any): any {
|
||||
const url: string = imageUrl.url;
|
||||
const m = /^data:([^;,]*);base64,(.*)$/s.exec(url);
|
||||
const mediaType: string | undefined =
|
||||
(m && m[1]) || (typeof imageUrl.mime_type === "string" ? imageUrl.mime_type : undefined);
|
||||
const fields = imageUrl.document && typeof imageUrl.document === "object" ? imageUrl.document : {};
|
||||
if (imageUrl.source && typeof imageUrl.source === "object") {
|
||||
return { type: "document", source: imageUrl.source, ...fields };
|
||||
}
|
||||
if (imageUrl.source_type === "text" && m) {
|
||||
return { type: "document", source: { type: "text", media_type: "text/plain", data: base64ToText(m[2]) }, ...fields };
|
||||
}
|
||||
if (mediaType === "application/pdf") {
|
||||
const source = m
|
||||
? { type: "base64", media_type: "application/pdf", data: m[2] }
|
||||
: { type: "url", url };
|
||||
return { type: "document", source, ...fields };
|
||||
}
|
||||
if (mediaType === undefined || mediaType.startsWith("image/")) return openAIImageBlock(url);
|
||||
return null;
|
||||
}
|
||||
|
||||
export function anthropicToOpenAI(messages: any[]): OpenAIMessage[] {
|
||||
const result: OpenAIMessage[] = [];
|
||||
|
||||
@@ -99,15 +174,15 @@ export function anthropicToOpenAI(messages: any[]): OpenAIMessage[] {
|
||||
}
|
||||
if (Array.isArray(msg.content)) {
|
||||
const toolResults = msg.content.filter((b: any) => b.type === "tool_result");
|
||||
const hasImages = msg.content.some((b: any) => b.type === "image");
|
||||
const hasMedia = msg.content.some((b: any) => b.type === "image" || b.type === "document");
|
||||
|
||||
if (hasImages) {
|
||||
if (hasMedia) {
|
||||
const parts = msg.content
|
||||
.filter((b: any) => b.type === "text" || b.type === "image")
|
||||
.filter((b: any) => b.type === "text" || b.type === "image" || b.type === "document")
|
||||
.map((b: any) => {
|
||||
if (b.type === "text") return { type: "text" as const, text: b.text };
|
||||
const url = anthropicImageUrl(b.source);
|
||||
return url ? { type: "image_url" as const, image_url: { url } } : null;
|
||||
const imageUrl = anthropicMediaImageUrl(b);
|
||||
return imageUrl ? { type: "image_url" as const, image_url: imageUrl } : null;
|
||||
})
|
||||
.filter((p: any): p is NonNullable<typeof p> => p !== null);
|
||||
|
||||
@@ -147,10 +222,22 @@ export function anthropicToOpenAI(messages: any[]): OpenAIMessage[] {
|
||||
if (Array.isArray(msg.content)) {
|
||||
const textBlocks = msg.content.filter((b: any) => b.type === "text");
|
||||
const toolUseBlocks = msg.content.filter((b: any) => b.type === "tool_use");
|
||||
const hasMedia = msg.content.some((b: any) => b.type === "image" || b.type === "document");
|
||||
|
||||
const content = textBlocks.length > 0
|
||||
? textBlocks.map((b: any) => b.text).join("\n")
|
||||
: null;
|
||||
let content: AssistantMessage["content"];
|
||||
if (hasMedia) {
|
||||
// Keep generated media as ordered parts (see AssistantMessage.content).
|
||||
content = msg.content
|
||||
.filter((b: any) => b.type === "text" || b.type === "image" || b.type === "document")
|
||||
.map((b: any) => {
|
||||
if (b.type === "text") return { type: "text" as const, text: b.text };
|
||||
const imageUrl = anthropicMediaImageUrl(b);
|
||||
return imageUrl ? { type: "image_url" as const, image_url: imageUrl } : null;
|
||||
})
|
||||
.filter((p: any): p is NonNullable<typeof p> => p !== null);
|
||||
} else {
|
||||
content = textBlocks.length > 0 ? textBlocks.map((b: any) => b.text).join("\n") : null;
|
||||
}
|
||||
|
||||
const openaiMsg: AssistantMessage = { role: "assistant", content };
|
||||
if (toolUseBlocks.length > 0) {
|
||||
@@ -190,7 +277,7 @@ export function openAIToAnthropic(messages: OpenAIMessage[]): any[] {
|
||||
role: "user",
|
||||
content: msg.content.map((p) => {
|
||||
if (p.type === "text") return { type: "text", text: p.text };
|
||||
if (p.type === "image_url") return openAIImageBlock(p.image_url.url);
|
||||
if (p.type === "image_url") return anthropicMediaBlock(p.image_url) ?? { type: "text", text: "" };
|
||||
return { type: "text", text: "" };
|
||||
}),
|
||||
});
|
||||
@@ -200,7 +287,17 @@ export function openAIToAnthropic(messages: OpenAIMessage[]): any[] {
|
||||
|
||||
if (msg.role === "assistant") {
|
||||
const blocks: any[] = [];
|
||||
if (msg.content) blocks.push({ type: "text", text: msg.content });
|
||||
if (typeof msg.content === "string") {
|
||||
if (msg.content) blocks.push({ type: "text", text: msg.content });
|
||||
} else if (Array.isArray(msg.content)) {
|
||||
for (const g of groupAssistantParts(msg.content)) {
|
||||
if ("text" in g) blocks.push({ type: "text", text: g.text });
|
||||
else {
|
||||
const block = anthropicMediaBlock(g.image_url);
|
||||
if (block) blocks.push(block);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (msg.tool_calls) {
|
||||
for (const tc of msg.tool_calls) {
|
||||
blocks.push({
|
||||
@@ -236,6 +333,167 @@ export function openAIToAnthropic(messages: OpenAIMessage[]): any[] {
|
||||
// Vercel AI SDK → OpenAI
|
||||
// ============================================================
|
||||
|
||||
function bytesToBase64(bytes: Uint8Array): string {
|
||||
if (typeof Buffer !== "undefined") return Buffer.from(bytes).toString("base64");
|
||||
let binary = "";
|
||||
for (let i = 0; i < bytes.length; i += 0x8000) {
|
||||
binary += String.fromCharCode(...bytes.subarray(i, i + 0x8000));
|
||||
}
|
||||
return btoa(binary);
|
||||
}
|
||||
|
||||
function parseUrl(value: string): URL | null {
|
||||
try {
|
||||
return new URL(value);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
type VercelMediaImageUrl = {
|
||||
url: string;
|
||||
mime_type?: string;
|
||||
part?: "file";
|
||||
filename?: string;
|
||||
// AI SDK 7 / LanguageModelV4 tagged file data: the tag the input used, so the reverse
|
||||
// conversion restores the same shape; absent for the bare AI SDK 6 forms.
|
||||
data_tag?: "data" | "url" | "text" | "reference";
|
||||
// A provider-managed file reference, carried verbatim (it has no bytes or url to convert).
|
||||
reference?: Record<string, string>;
|
||||
};
|
||||
|
||||
function base64ToText(b64: string): string {
|
||||
if (typeof Buffer !== "undefined") return Buffer.from(b64, "base64").toString("utf8");
|
||||
const binary = atob(b64);
|
||||
const bytes = new Uint8Array(binary.length);
|
||||
for (let i = 0; i < binary.length; i++) bytes[i] = binary.charCodeAt(i);
|
||||
return new TextDecoder().decode(bytes);
|
||||
}
|
||||
|
||||
// A bare AI SDK 6 provider reference: a plain string-valued record with no `type`.
|
||||
function isProviderReference(v: any): boolean {
|
||||
return (
|
||||
!!v &&
|
||||
typeof v === "object" &&
|
||||
!Array.isArray(v) &&
|
||||
!(v instanceof Uint8Array) &&
|
||||
!(v instanceof ArrayBuffer) &&
|
||||
!(v instanceof URL) &&
|
||||
v.type === undefined &&
|
||||
Object.values(v).every((x) => typeof x === "string")
|
||||
);
|
||||
}
|
||||
|
||||
// Vercel `image` / `file` part -> OpenAI image_url object, or null if the data is unconvertible.
|
||||
// Accepts the bare AI SDK 6 forms (bytes, base64, data URL, URL, provider reference) and the
|
||||
// AI SDK 7 / LanguageModelV4 tagged forms ({ type: "data" | "url" | "text" | "reference" }).
|
||||
// Bytes, base64 and text travel as a data: URI, which carries the media type; URLs stay URLs
|
||||
// with the media type on a private `mime_type` key; references are carried verbatim on
|
||||
// `reference`. `part: "file"`, `filename` and `data_tag` are carried so the reverse conversion
|
||||
// rebuilds the right part in the same shape. The proxy passes non-text parts through unchanged.
|
||||
function vercelMediaImageUrl(p: any): VercelMediaImageUrl | null {
|
||||
let data = p.type === "file" ? p.data : p.image;
|
||||
const mediaType: string | undefined = typeof p.mediaType === "string" ? p.mediaType : p.mimeType;
|
||||
const out: VercelMediaImageUrl = { url: "" };
|
||||
if (p.type === "file") out.part = "file";
|
||||
if (typeof p.filename === "string") out.filename = p.filename;
|
||||
|
||||
if (data && typeof data === "object" && typeof data.type === "string") {
|
||||
if (data.type === "data") {
|
||||
out.data_tag = "data";
|
||||
data = data.data;
|
||||
} else if (data.type === "url") {
|
||||
out.data_tag = "url";
|
||||
data = data.url;
|
||||
} else if (data.type === "text" && typeof data.text === "string") {
|
||||
out.data_tag = "text";
|
||||
out.url = `data:${mediaType ?? "text/plain"};base64,${bytesToBase64(new TextEncoder().encode(data.text))}`;
|
||||
return out;
|
||||
} else if (data.type === "reference" && data.reference && typeof data.reference === "object") {
|
||||
out.data_tag = "reference";
|
||||
out.reference = data.reference;
|
||||
out.url = `data:${mediaType ?? ""};base64,`;
|
||||
return out;
|
||||
} else {
|
||||
return null;
|
||||
}
|
||||
} else if (isProviderReference(data)) {
|
||||
out.reference = data;
|
||||
out.url = `data:${mediaType ?? ""};base64,`;
|
||||
return out;
|
||||
}
|
||||
|
||||
let carryType = false;
|
||||
if (data instanceof Uint8Array || data instanceof ArrayBuffer) {
|
||||
const bytes = data instanceof Uint8Array ? data : new Uint8Array(data);
|
||||
out.url = `data:${mediaType ?? ""};base64,${bytesToBase64(bytes)}`;
|
||||
} else if (data instanceof URL) {
|
||||
out.url = data.toString();
|
||||
carryType = true;
|
||||
} else if (typeof data === "string") {
|
||||
if (data.startsWith("data:")) out.url = data;
|
||||
else if (parseUrl(data)) {
|
||||
out.url = data;
|
||||
carryType = true;
|
||||
} else out.url = `data:${mediaType ?? ""};base64,${data}`;
|
||||
} else {
|
||||
return null;
|
||||
}
|
||||
if (carryType && mediaType) out.mime_type = mediaType;
|
||||
return out;
|
||||
}
|
||||
|
||||
// OpenAI image_url -> Vercel `image` or `file` part, in the shape the forward conversion saw
|
||||
// (bare, or LanguageModelV4-tagged when `data_tag` is carried). A data: URI yields base64 data
|
||||
// plus its media type; an http(s) url yields a URL object; anything else stays a string; a
|
||||
// carried provider reference comes back verbatim. Never throws.
|
||||
function vercelMediaPart(imageUrl: any): any {
|
||||
const url: string = imageUrl.url;
|
||||
const tag: string | undefined = typeof imageUrl.data_tag === "string" ? imageUrl.data_tag : undefined;
|
||||
const m = /^data:([^;,]*);base64,(.*)$/s.exec(url);
|
||||
let mediaType: string | undefined = typeof imageUrl.mime_type === "string" ? imageUrl.mime_type : undefined;
|
||||
if (m && m[1]) mediaType = m[1];
|
||||
let data: any;
|
||||
if (imageUrl.reference && typeof imageUrl.reference === "object") {
|
||||
data = tag === "reference" ? { type: "reference", reference: imageUrl.reference } : imageUrl.reference;
|
||||
} else if (m) {
|
||||
data = m[2];
|
||||
if (tag === "data") data = { type: "data", data: m[2] };
|
||||
else if (tag === "text") data = { type: "text", text: base64ToText(m[2]) };
|
||||
} else {
|
||||
const parsed = parseUrl(url);
|
||||
data = parsed ?? url;
|
||||
if (tag === "url") data = { type: "url", url: parsed ?? url };
|
||||
}
|
||||
const isFile = imageUrl.part === "file" || (mediaType !== undefined && !mediaType.startsWith("image/"));
|
||||
if (isFile) {
|
||||
const part: any = { type: "file", mediaType: mediaType ?? "application/octet-stream", data };
|
||||
if (typeof imageUrl.filename === "string") part.filename = imageUrl.filename;
|
||||
return part;
|
||||
}
|
||||
const part: any = { type: "image", image: data };
|
||||
if (mediaType) part.mediaType = mediaType;
|
||||
return part;
|
||||
}
|
||||
|
||||
// Assistant content parts for the Anthropic / Gemini reverse converters. A text-only array
|
||||
// collapses to one text joined with "\n" (the forward converters join text-only turns the same
|
||||
// way); an array carrying media keeps every part separate, because the forward converters keep
|
||||
// the exact parts in that case and a round trip must not add or move characters.
|
||||
function groupAssistantParts(parts: ContentPart[]): Array<{ text: string } | { image_url: any }> {
|
||||
const hasMedia = parts.some((p) => p.type === "image_url");
|
||||
if (!hasMedia) {
|
||||
const text = parts.filter((p) => p.type === "text").map((p) => (p as any).text).join("\n");
|
||||
return text ? [{ text }] : [];
|
||||
}
|
||||
const out: Array<{ text: string } | { image_url: any }> = [];
|
||||
for (const p of parts) {
|
||||
if (p.type === "text") out.push({ text: p.text });
|
||||
else if (p.type === "image_url") out.push({ image_url: p.image_url });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export function vercelToOpenAI(messages: any[]): OpenAIMessage[] {
|
||||
const result: OpenAIMessage[] = [];
|
||||
|
||||
@@ -252,21 +510,19 @@ export function vercelToOpenAI(messages: any[]): OpenAIMessage[] {
|
||||
}
|
||||
const parts = Array.isArray(msg.content) ? msg.content : [];
|
||||
const textParts = parts.filter((p: any) => p.type === "text");
|
||||
const imageParts = parts.filter((p: any) => p.type === "image");
|
||||
const mediaParts = parts.filter((p: any) => p.type === "image" || p.type === "file");
|
||||
|
||||
if (imageParts.length === 0 && textParts.length > 0) {
|
||||
if (mediaParts.length === 0 && textParts.length > 0) {
|
||||
result.push({ role: "user", content: textParts.map((p: any) => p.text).join("") });
|
||||
} else {
|
||||
const openaiParts = parts
|
||||
.filter((p: any) => p.type === "text" || p.type === "image")
|
||||
.filter((p: any) => p.type === "text" || p.type === "image" || p.type === "file")
|
||||
.map((p: any) => {
|
||||
if (p.type === "text") return { type: "text" as const, text: p.text };
|
||||
if (p.type === "image") {
|
||||
const url = p.image instanceof URL ? p.image.toString() : String(p.image);
|
||||
return { type: "image_url" as const, image_url: { url } };
|
||||
}
|
||||
return { type: "text" as const, text: "" };
|
||||
});
|
||||
const imageUrl = vercelMediaImageUrl(p);
|
||||
return imageUrl ? { type: "image_url" as const, image_url: imageUrl } : null;
|
||||
})
|
||||
.filter((p: any): p is NonNullable<typeof p> => p !== null);
|
||||
result.push({ role: "user", content: openaiParts });
|
||||
}
|
||||
continue;
|
||||
@@ -280,8 +536,22 @@ export function vercelToOpenAI(messages: any[]): OpenAIMessage[] {
|
||||
const parts = Array.isArray(msg.content) ? msg.content : [];
|
||||
const textParts = parts.filter((p: any) => p.type === "text");
|
||||
const toolCallParts = parts.filter((p: any) => p.type === "tool-call");
|
||||
const hasFiles = parts.some((p: any) => p.type === "file");
|
||||
|
||||
const content = textParts.length > 0 ? textParts.map((p: any) => p.text).join("") : null;
|
||||
let content: AssistantMessage["content"];
|
||||
if (hasFiles) {
|
||||
// AI SDK assistant content may carry generated files; keep them as ordered parts.
|
||||
content = parts
|
||||
.filter((p: any) => p.type === "text" || p.type === "file")
|
||||
.map((p: any) => {
|
||||
if (p.type === "text") return { type: "text" as const, text: p.text };
|
||||
const imageUrl = vercelMediaImageUrl(p);
|
||||
return imageUrl ? { type: "image_url" as const, image_url: imageUrl } : null;
|
||||
})
|
||||
.filter((p: any): p is NonNullable<typeof p> => p !== null);
|
||||
} else {
|
||||
content = textParts.length > 0 ? textParts.map((p: any) => p.text).join("") : null;
|
||||
}
|
||||
const openaiMsg: AssistantMessage = { role: "assistant", content };
|
||||
|
||||
if (toolCallParts.length > 0) {
|
||||
@@ -343,7 +613,7 @@ export function openAIToVercel(messages: OpenAIMessage[]): any[] {
|
||||
} else if (Array.isArray(msg.content)) {
|
||||
const parts = msg.content.map((p) => {
|
||||
if (p.type === "text") return { type: "text", text: p.text };
|
||||
if (p.type === "image_url") return { type: "image", image: new URL(p.image_url.url) };
|
||||
if (p.type === "image_url") return vercelMediaPart(p.image_url);
|
||||
return { type: "text", text: "" };
|
||||
});
|
||||
result.push({ role: "user", content: parts });
|
||||
@@ -353,7 +623,14 @@ export function openAIToVercel(messages: OpenAIMessage[]): any[] {
|
||||
|
||||
if (msg.role === "assistant") {
|
||||
const parts: any[] = [];
|
||||
if (msg.content) parts.push({ type: "text", text: msg.content });
|
||||
if (typeof msg.content === "string") {
|
||||
if (msg.content) parts.push({ type: "text", text: msg.content });
|
||||
} else if (Array.isArray(msg.content)) {
|
||||
for (const p of msg.content) {
|
||||
if (p.type === "text") parts.push({ type: "text", text: p.text });
|
||||
else if (p.type === "image_url") parts.push(vercelMediaPart(p.image_url));
|
||||
}
|
||||
}
|
||||
if (msg.tool_calls) {
|
||||
for (const tc of msg.tool_calls) {
|
||||
toolNames.set(tc.id, tc.function.name);
|
||||
@@ -480,8 +757,22 @@ export function geminiToOpenAI(messages: any[]): OpenAIMessage[] {
|
||||
if (role === "assistant") {
|
||||
const textParts = parts.filter((p: any) => p.text !== undefined);
|
||||
const funcCalls = parts.filter((p: any) => p.functionCall);
|
||||
const hasMedia = parts.some((p: any) => p.inlineData || p.fileData);
|
||||
|
||||
const content = textParts.length > 0 ? textParts.map((p: any) => p.text).join("\n") : null;
|
||||
let content: AssistantMessage["content"];
|
||||
if (hasMedia) {
|
||||
// Keep generated media (e.g. image output) as ordered parts.
|
||||
content = parts
|
||||
.filter((p: any) => p.text !== undefined || p.inlineData || p.fileData)
|
||||
.map((p: any) => {
|
||||
if (p.text !== undefined) return { type: "text" as const, text: p.text };
|
||||
const imageUrl = geminiMediaImageUrl(p);
|
||||
return imageUrl ? { type: "image_url" as const, image_url: imageUrl } : null;
|
||||
})
|
||||
.filter((p: any): p is NonNullable<typeof p> => p !== null);
|
||||
} else {
|
||||
content = textParts.length > 0 ? textParts.map((p: any) => p.text).join("\n") : null;
|
||||
}
|
||||
const openaiMsg: AssistantMessage = { role: "assistant", content };
|
||||
|
||||
if (funcCalls.length > 0) {
|
||||
@@ -533,7 +824,14 @@ export function openAIToGemini(messages: OpenAIMessage[]): any[] {
|
||||
|
||||
if (msg.role === "assistant") {
|
||||
const parts: any[] = [];
|
||||
if (msg.content) parts.push({ text: msg.content });
|
||||
if (typeof msg.content === "string") {
|
||||
if (msg.content) parts.push({ text: msg.content });
|
||||
} else if (Array.isArray(msg.content)) {
|
||||
for (const g of groupAssistantParts(msg.content)) {
|
||||
if ("text" in g) parts.push({ text: g.text });
|
||||
else parts.push(geminiMediaPart(g.image_url));
|
||||
}
|
||||
}
|
||||
if (msg.tool_calls) {
|
||||
for (const tc of msg.tool_calls) {
|
||||
parts.push({
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { describe, it, expect, vi, beforeEach } from "vitest";
|
||||
import type { LanguageModelV3 } from "@ai-sdk/provider";
|
||||
import type { LanguageModelV3, LanguageModelV4FilePart, LanguageModelV4Prompt } from "@ai-sdk/provider";
|
||||
import {
|
||||
headroomMiddleware,
|
||||
compressVercelMessages,
|
||||
@@ -134,6 +134,149 @@ describe("headroomMiddleware", () => {
|
||||
const body = JSON.parse(mockFetch.mock.calls[0][1].body);
|
||||
expect(body.model).toBe("claude-sonnet-4-5-20250929");
|
||||
});
|
||||
|
||||
it("keeps a provider-prompt file part through compression", async () => {
|
||||
// The proxy only compresses text and passes the media part back untouched.
|
||||
mockFetch.mockResolvedValueOnce(
|
||||
mockCompressResponse([
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "compressed" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file" } },
|
||||
],
|
||||
},
|
||||
]),
|
||||
);
|
||||
const middleware = headroomMiddleware({ baseUrl: "http://localhost:8787" });
|
||||
const out = await middleware.transformParams({
|
||||
params: {
|
||||
prompt: [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "what is this?" },
|
||||
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]) },
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
model: {},
|
||||
type: "generate",
|
||||
});
|
||||
|
||||
const body = JSON.parse(mockFetch.mock.calls[0][1].body);
|
||||
expect(body.messages[0].content[1]).toEqual({
|
||||
type: "image_url",
|
||||
image_url: { url: "data:image/png;base64,iVBORw==", part: "file" },
|
||||
});
|
||||
expect(out.prompt[0].content[1]).toEqual({ type: "file", mediaType: "image/png", data: "iVBORw==" });
|
||||
});
|
||||
|
||||
it("keeps an assistant file part (generated media) through compression", async () => {
|
||||
mockFetch.mockResolvedValueOnce(
|
||||
mockCompressResponse([
|
||||
{ role: "user", content: "draw a cat" },
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file" } },
|
||||
],
|
||||
},
|
||||
]),
|
||||
);
|
||||
const middleware = headroomMiddleware({ baseUrl: "http://localhost:8787" });
|
||||
const out = await middleware.transformParams({
|
||||
params: {
|
||||
prompt: [
|
||||
{ role: "user", content: [{ type: "text", text: "draw a cat" }] },
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]) },
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
model: {},
|
||||
type: "generate",
|
||||
});
|
||||
|
||||
const body = JSON.parse(mockFetch.mock.calls[0][1].body);
|
||||
expect(body.messages[1]).toEqual({
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file" } },
|
||||
],
|
||||
});
|
||||
expect(out.prompt[1]).toEqual({
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "file", mediaType: "image/png", data: "iVBORw==" },
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("round-trips a LanguageModelV4 file part (tagged data) through transformParams", async () => {
|
||||
const filePart: LanguageModelV4FilePart = {
|
||||
type: "file",
|
||||
mediaType: "image/png",
|
||||
data: { type: "data", data: new Uint8Array([137, 80, 78, 71]) },
|
||||
};
|
||||
const prompt: LanguageModelV4Prompt = [{ role: "user", content: [{ type: "text", text: "what is this?" }, filePart] }];
|
||||
// The proxy keeps the media part and compresses only the text.
|
||||
mockFetch.mockResolvedValueOnce(
|
||||
mockCompressResponse([
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "compressed" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file", data_tag: "data" } },
|
||||
],
|
||||
},
|
||||
]),
|
||||
);
|
||||
const middleware = headroomMiddleware({ baseUrl: "http://localhost:8787" });
|
||||
const out = await middleware.transformParams({ params: { prompt }, model: { specificationVersion: "v4" }, type: "generate" });
|
||||
|
||||
const body = JSON.parse(mockFetch.mock.calls[0][1].body);
|
||||
expect(body.messages[0].content[1]).toEqual({
|
||||
type: "image_url",
|
||||
image_url: { url: "data:image/png;base64,iVBORw==", part: "file", data_tag: "data" },
|
||||
});
|
||||
const restored: LanguageModelV4FilePart = out.prompt[0].content[1];
|
||||
expect(restored).toEqual({ type: "file", mediaType: "image/png", data: { type: "data", data: "iVBORw==" } });
|
||||
});
|
||||
|
||||
it("passes a provider file reference through transformParams untouched", async () => {
|
||||
const filePart: LanguageModelV4FilePart = {
|
||||
type: "file",
|
||||
mediaType: "application/pdf",
|
||||
data: { type: "reference", reference: { openai: "file-abc" } },
|
||||
};
|
||||
mockFetch.mockResolvedValueOnce(
|
||||
mockCompressResponse([
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "compressed" },
|
||||
{ type: "image_url", image_url: { url: "data:application/pdf;base64,", part: "file", data_tag: "reference", reference: { openai: "file-abc" } } },
|
||||
],
|
||||
},
|
||||
]),
|
||||
);
|
||||
const middleware = headroomMiddleware({ baseUrl: "http://localhost:8787" });
|
||||
const out = await middleware.transformParams({
|
||||
params: { prompt: [{ role: "user", content: [{ type: "text", text: "summarise" }, filePart] }] },
|
||||
model: { specificationVersion: "v4" },
|
||||
type: "generate",
|
||||
});
|
||||
expect(out.prompt[0].content[1]).toEqual(filePart);
|
||||
});
|
||||
});
|
||||
|
||||
describe("compressVercelMessages", () => {
|
||||
@@ -205,6 +348,22 @@ describe("compressVercelMessages", () => {
|
||||
|
||||
expect(result.messages[1].content[0].toolName).toBe("getWeather");
|
||||
});
|
||||
|
||||
it("does not throw on byte image data", async () => {
|
||||
mockFetch.mockResolvedValueOnce(
|
||||
mockCompressResponse([
|
||||
{ role: "user", content: [{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } }] },
|
||||
]),
|
||||
);
|
||||
await expect(
|
||||
compressVercelMessages(
|
||||
[{ role: "user", content: [{ type: "image", image: new Uint8Array([137, 80, 78, 71]), mediaType: "image/png" }] }],
|
||||
{ baseUrl: "http://localhost:8787" },
|
||||
),
|
||||
).resolves.toMatchObject({
|
||||
messages: [{ role: "user", content: [{ type: "image", image: "iVBORw==", mediaType: "image/png" }] }],
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("withHeadroom", () => {
|
||||
|
||||
@@ -135,3 +135,70 @@ describe("compress()", () => {
|
||||
expect(body.config.target_ratio).toBe(0.25);
|
||||
});
|
||||
});
|
||||
|
||||
describe("compress() autodetects Vercel media-only conversations", () => {
|
||||
beforeEach(() => {
|
||||
mockFetch.mockReset();
|
||||
});
|
||||
|
||||
it("sends binary file parts as image_url data URIs and restores Vercel file parts", async () => {
|
||||
mockFetch.mockResolvedValueOnce(
|
||||
okResponse({
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "compressed" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file" } },
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
|
||||
const result = await compress(
|
||||
[
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "what is this?" },
|
||||
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]) },
|
||||
],
|
||||
},
|
||||
],
|
||||
{ baseUrl: "http://localhost:8787" },
|
||||
);
|
||||
|
||||
const body = JSON.parse(mockFetch.mock.calls[0][1].body);
|
||||
expect(body.messages[0].content[1]).toEqual({
|
||||
type: "image_url",
|
||||
image_url: { url: "data:image/png;base64,iVBORw==", part: "file" },
|
||||
});
|
||||
expect(JSON.stringify(body)).not.toContain('"0":137');
|
||||
expect(result.messages[0].content[1]).toEqual({ type: "file", mediaType: "image/png", data: "iVBORw==" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("compress() keeps Anthropic text documents through the public path", () => {
|
||||
beforeEach(() => {
|
||||
mockFetch.mockReset();
|
||||
});
|
||||
|
||||
it("sends a document-only turn as a media part and restores the document block", async () => {
|
||||
const doc = {
|
||||
type: "document",
|
||||
source: { type: "text", media_type: "text/plain", data: "Critical contract: do not delete account 123" },
|
||||
title: "Contract",
|
||||
citations: { enabled: true },
|
||||
};
|
||||
// The proxy passes non-text parts through unchanged; echo the request's messages back.
|
||||
mockFetch.mockImplementationOnce(async (_url: string, init: any) => okResponse({ messages: JSON.parse(init.body).messages }));
|
||||
|
||||
const result = await compress([{ role: "user", content: [doc] }], { baseUrl: "http://localhost:8787" });
|
||||
|
||||
const body = JSON.parse(mockFetch.mock.calls[0][1].body);
|
||||
expect(body.messages).toHaveLength(1);
|
||||
expect(body.messages[0].content[0].type).toBe("image_url");
|
||||
expect(result.messages).toEqual([{ role: "user", content: [doc] }]);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -66,9 +66,9 @@ describe("Message types", () => {
|
||||
>();
|
||||
});
|
||||
|
||||
it("AssistantMessage content can be string or null", () => {
|
||||
it("AssistantMessage content can be a string, content parts, or null", () => {
|
||||
expectTypeOf<AssistantMessage["role"]>().toEqualTypeOf<"assistant">();
|
||||
expectTypeOf<AssistantMessage["content"]>().toEqualTypeOf<string | null>();
|
||||
expectTypeOf<AssistantMessage["content"]>().toEqualTypeOf<string | ContentPart[] | null>();
|
||||
});
|
||||
|
||||
it("AssistantMessage tool_calls is optional", () => {
|
||||
|
||||
@@ -131,3 +131,57 @@ describe("detectFormat", () => {
|
||||
expect(detectFormat(messages)).toBe("vercel");
|
||||
});
|
||||
});
|
||||
|
||||
describe("detectFormat: Vercel media parts", () => {
|
||||
it("detects a media-only Vercel turn with a file part", () => {
|
||||
expect(
|
||||
detectFormat([
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "what is this?" },
|
||||
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]) },
|
||||
],
|
||||
},
|
||||
]),
|
||||
).toBe("vercel");
|
||||
});
|
||||
|
||||
it("detects a media-only Vercel turn with an image part", () => {
|
||||
expect(
|
||||
detectFormat([
|
||||
{ role: "user", content: [{ type: "image", image: new Uint8Array([137, 80, 78, 71]), mediaType: "image/png" }] },
|
||||
]),
|
||||
).toBe("vercel");
|
||||
});
|
||||
|
||||
it("still treats OpenAI image_url parts as openai", () => {
|
||||
expect(
|
||||
detectFormat([{ role: "user", content: [{ type: "image_url", image_url: { url: "https://example.com/a.png" } }] }]),
|
||||
).toBe("openai");
|
||||
});
|
||||
|
||||
it("still treats Anthropic image blocks (with a source) as anthropic", () => {
|
||||
expect(
|
||||
detectFormat([
|
||||
{ role: "user", content: [{ type: "image", source: { type: "base64", media_type: "image/png", data: "AAAA" } }] },
|
||||
]),
|
||||
).toBe("anthropic");
|
||||
});
|
||||
});
|
||||
|
||||
describe("detectFormat: Anthropic document blocks", () => {
|
||||
it("detects a turn whose only Anthropic marker is a document block", () => {
|
||||
expect(
|
||||
detectFormat([
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "Report" },
|
||||
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: "JVBERi0=" } },
|
||||
],
|
||||
},
|
||||
]),
|
||||
).toBe("anthropic");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -6,6 +6,9 @@ import {
|
||||
openAIToAnthropic,
|
||||
geminiToOpenAI,
|
||||
openAIToGemini,
|
||||
fromOpenAI,
|
||||
detectFormat,
|
||||
toOpenAI,
|
||||
} from "../../src/utils/format.js";
|
||||
import type { OpenAIMessage } from "../../src/types.js";
|
||||
|
||||
@@ -838,3 +841,621 @@ describe("openAIToVercel tool names", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("vercelToOpenAI media parts", () => {
|
||||
it("converts a provider-prompt file part (image bytes) to an image_url data URI", () => {
|
||||
const result = vercelToOpenAI([
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "what is this?" },
|
||||
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]) },
|
||||
],
|
||||
},
|
||||
]);
|
||||
expect(result).toEqual([
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "what is this?" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps a file-only turn (pdf) and carries its filename", () => {
|
||||
const result = vercelToOpenAI([
|
||||
{ role: "user", content: [{ type: "file", mediaType: "application/pdf", data: "JVBERi0=", filename: "a.pdf" }] },
|
||||
]);
|
||||
expect(result).toEqual([
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "image_url", image_url: { url: "data:application/pdf;base64,JVBERi0=", part: "file", filename: "a.pdf" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("encodes a Uint8Array image as a data URI instead of String(bytes)", () => {
|
||||
const result = vercelToOpenAI([
|
||||
{ role: "user", content: [{ type: "image", image: new Uint8Array([137, 80, 78, 71]), mediaType: "image/png" }] },
|
||||
]);
|
||||
expect(result).toEqual([
|
||||
{ role: "user", content: [{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } }] },
|
||||
]);
|
||||
});
|
||||
|
||||
it("treats a bare base64 string image as data, not a URL", () => {
|
||||
const result = vercelToOpenAI([
|
||||
{ role: "user", content: [{ type: "image", image: "iVBORw==", mediaType: "image/png" }] },
|
||||
]);
|
||||
expect(result).toEqual([
|
||||
{ role: "user", content: [{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } }] },
|
||||
]);
|
||||
});
|
||||
|
||||
it("carries the MIME type of a URL-backed file part", () => {
|
||||
const result = vercelToOpenAI([
|
||||
{ role: "user", content: [{ type: "file", mediaType: "application/pdf", data: new URL("https://example.com/a.pdf") }] },
|
||||
]);
|
||||
expect(result).toEqual([
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "image_url", image_url: { url: "https://example.com/a.pdf", part: "file", mime_type: "application/pdf" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("openAIToVercel media parts", () => {
|
||||
it("restores a data URI marked part=file as a file part with base64 data", () => {
|
||||
const msgs = [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "image_url", image_url: { url: "data:application/pdf;base64,JVBERi0=", part: "file", filename: "a.pdf" } },
|
||||
],
|
||||
},
|
||||
] as OpenAIMessage[];
|
||||
expect(openAIToVercel(msgs)).toEqual([
|
||||
{ role: "user", content: [{ type: "file", mediaType: "application/pdf", data: "JVBERi0=", filename: "a.pdf" }] },
|
||||
]);
|
||||
});
|
||||
|
||||
it("restores an image data URI as an image part with base64 data and mediaType", () => {
|
||||
const msgs: OpenAIMessage[] = [
|
||||
{ role: "user", content: [{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } }] },
|
||||
];
|
||||
expect(openAIToVercel(msgs)).toEqual([
|
||||
{ role: "user", content: [{ type: "image", image: "iVBORw==", mediaType: "image/png" }] },
|
||||
]);
|
||||
});
|
||||
|
||||
it("restores a URL-backed file part from the carried MIME type", () => {
|
||||
const msgs = [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "image_url", image_url: { url: "https://example.com/a.pdf", part: "file", mime_type: "application/pdf" } },
|
||||
],
|
||||
},
|
||||
] as OpenAIMessage[];
|
||||
const part = openAIToVercel(msgs)[0].content[0];
|
||||
expect(part.type).toBe("file");
|
||||
expect(part.mediaType).toBe("application/pdf");
|
||||
expect(part.data).toBeInstanceOf(URL);
|
||||
expect(part.data.toString()).toBe("https://example.com/a.pdf");
|
||||
});
|
||||
|
||||
it("never throws on an unparseable url and keeps it as a string", () => {
|
||||
const msgs: OpenAIMessage[] = [
|
||||
{ role: "user", content: [{ type: "image_url", image_url: { url: "137,80,78,71" } }] },
|
||||
];
|
||||
expect(() => openAIToVercel(msgs)).not.toThrow();
|
||||
expect(openAIToVercel(msgs)[0].content[0]).toEqual({ type: "image", image: "137,80,78,71" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("round-trip: vercelToOpenAI then openAIToVercel (media)", () => {
|
||||
it("round-trips a provider-prompt text+file turn into a valid file part", () => {
|
||||
const original = [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "what is this?" },
|
||||
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]) },
|
||||
],
|
||||
},
|
||||
];
|
||||
expect(openAIToVercel(vercelToOpenAI(original))).toEqual([
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "what is this?" },
|
||||
{ type: "file", mediaType: "image/png", data: "iVBORw==" },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("round-trips a byte image without throwing", () => {
|
||||
const original = [
|
||||
{ role: "user", content: [{ type: "image", image: new Uint8Array([137, 80, 78, 71]), mediaType: "image/png" }] },
|
||||
];
|
||||
expect(openAIToVercel(vercelToOpenAI(original))).toEqual([
|
||||
{ role: "user", content: [{ type: "image", image: "iVBORw==", mediaType: "image/png" }] },
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("vercelToOpenAI / openAIToVercel assistant file parts", () => {
|
||||
it("keeps an assistant file part (generated image) alongside its text", () => {
|
||||
const result = vercelToOpenAI([
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]), filename: "cat.png" },
|
||||
],
|
||||
},
|
||||
]);
|
||||
expect(result).toEqual([
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file", filename: "cat.png" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("restores an assistant content array as text and file parts with the assistant role", () => {
|
||||
const msgs = [
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file", filename: "cat.png" } },
|
||||
],
|
||||
},
|
||||
] as OpenAIMessage[];
|
||||
expect(openAIToVercel(msgs)).toEqual([
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "file", mediaType: "image/png", data: "iVBORw==", filename: "cat.png" },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("round-trips an assistant text+file turn", () => {
|
||||
const original = [
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]) },
|
||||
],
|
||||
},
|
||||
];
|
||||
expect(openAIToVercel(vercelToOpenAI(original))).toEqual([
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "file", mediaType: "image/png", data: "iVBORw==" },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("leaves text-only assistant turns as a string (backward compat)", () => {
|
||||
expect(
|
||||
vercelToOpenAI([{ role: "assistant", content: [{ type: "text", text: "a" }, { type: "text", text: "b" }] }]),
|
||||
).toEqual([{ role: "assistant", content: "ab" }]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("assistant media across formats", () => {
|
||||
it("openAIToAnthropic keeps an assistant image as an image block after the text", () => {
|
||||
const msgs = [
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } },
|
||||
],
|
||||
},
|
||||
] as OpenAIMessage[];
|
||||
expect(openAIToAnthropic(msgs)).toEqual([
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw==" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("openAIToAnthropic maps an assistant PDF to a document block (base64 and url sources)", () => {
|
||||
const msgs = [
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "image_url", image_url: { url: "data:application/pdf;base64,JVBERi0=", part: "file" } },
|
||||
{ type: "image_url", image_url: { url: "https://example.com/a.pdf", part: "file", mime_type: "application/pdf" } },
|
||||
],
|
||||
},
|
||||
] as OpenAIMessage[];
|
||||
expect(openAIToAnthropic(msgs)).toEqual([
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: "JVBERi0=" } },
|
||||
{ type: "document", source: { type: "url", url: "https://example.com/a.pdf" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("openAIToAnthropic leaves a text-only assistant array as a string (backward compat)", () => {
|
||||
const msgs = [{ role: "assistant", content: [{ type: "text", text: "a" }] }] as OpenAIMessage[];
|
||||
expect(openAIToAnthropic(msgs)).toEqual([{ role: "assistant", content: "a" }]);
|
||||
});
|
||||
|
||||
it("openAIToGemini keeps an assistant image as an inlineData part after the text", () => {
|
||||
const msgs = [
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } },
|
||||
],
|
||||
},
|
||||
] as OpenAIMessage[];
|
||||
expect(openAIToGemini(msgs)).toEqual([
|
||||
{ role: "model", parts: [{ text: "generated image" }, { inlineData: { mimeType: "image/png", data: "iVBORw==" } }] },
|
||||
]);
|
||||
});
|
||||
|
||||
it("openAIToGemini maps a URL-backed assistant file to fileData with its MIME type", () => {
|
||||
const msgs = [
|
||||
{
|
||||
role: "assistant",
|
||||
content: [{ type: "image_url", image_url: { url: "https://example.com/a.pdf", part: "file", mime_type: "application/pdf" } }],
|
||||
},
|
||||
] as OpenAIMessage[];
|
||||
expect(openAIToGemini(msgs)).toEqual([
|
||||
{ role: "model", parts: [{ fileData: { mimeType: "application/pdf", fileUri: "https://example.com/a.pdf" } }] },
|
||||
]);
|
||||
});
|
||||
|
||||
it("anthropicToOpenAI keeps assistant image and document blocks as ordered parts", () => {
|
||||
const result = anthropicToOpenAI([
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw==" } },
|
||||
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: "JVBERi0=" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
expect(result).toEqual([
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } },
|
||||
{ type: "image_url", image_url: { url: "data:application/pdf;base64,JVBERi0=" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("geminiToOpenAI keeps model-turn inlineData as an ordered image_url part", () => {
|
||||
const result = geminiToOpenAI([
|
||||
{ role: "model", parts: [{ text: "generated image" }, { inlineData: { mimeType: "image/png", data: "iVBORw==" } }] },
|
||||
]);
|
||||
expect(result).toEqual([
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("round-trips an Anthropic assistant text+image turn exactly", () => {
|
||||
const original = [
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw==" } },
|
||||
],
|
||||
},
|
||||
];
|
||||
expect(openAIToAnthropic(anthropicToOpenAI(original))).toEqual(original);
|
||||
});
|
||||
|
||||
it("round-trips a Gemini model text+inlineData turn exactly", () => {
|
||||
const original = [
|
||||
{ role: "model", parts: [{ text: "generated image" }, { inlineData: { mimeType: "image/png", data: "iVBORw==" } }] },
|
||||
];
|
||||
expect(openAIToGemini(geminiToOpenAI(original))).toEqual(original);
|
||||
});
|
||||
|
||||
it("carries a Vercel assistant file into Gemini and Anthropic media parts", () => {
|
||||
const openai = vercelToOpenAI([
|
||||
{ role: "assistant", content: [{ type: "text", text: "generated image" }, { type: "file", mediaType: "image/png", data: "iVBORw==" }] },
|
||||
]);
|
||||
expect(fromOpenAI(openai, "gemini")).toEqual([
|
||||
{ role: "model", parts: [{ text: "generated image" }, { inlineData: { mimeType: "image/png", data: "iVBORw==" } }] },
|
||||
]);
|
||||
expect(fromOpenAI(openai, "anthropic")).toEqual([
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "generated image" },
|
||||
{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw==" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Vercel V4 tagged file data", () => {
|
||||
it("carries a data-tagged file part and restores the tagged shape", () => {
|
||||
const original = [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "what is this?" },
|
||||
{ type: "file", mediaType: "image/png", data: { type: "data", data: new Uint8Array([137, 80, 78, 71]) } },
|
||||
],
|
||||
},
|
||||
];
|
||||
const openai = vercelToOpenAI(original);
|
||||
expect(openai[0].content[1]).toEqual({
|
||||
type: "image_url",
|
||||
image_url: { url: "data:image/png;base64,iVBORw==", part: "file", data_tag: "data" },
|
||||
});
|
||||
expect(openAIToVercel(openai)).toEqual([
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "what is this?" },
|
||||
{ type: "file", mediaType: "image/png", data: { type: "data", data: "iVBORw==" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("carries a url-tagged file part and restores a URL object inside the tag", () => {
|
||||
const original = [
|
||||
{ role: "user", content: [{ type: "file", mediaType: "application/pdf", data: { type: "url", url: new URL("https://example.com/a.pdf") } }] },
|
||||
];
|
||||
const openai = vercelToOpenAI(original);
|
||||
expect(openai[0].content[0]).toEqual({
|
||||
type: "image_url",
|
||||
image_url: { url: "https://example.com/a.pdf", part: "file", mime_type: "application/pdf", data_tag: "url" },
|
||||
});
|
||||
const part = openAIToVercel(openai)[0].content[0];
|
||||
expect(part.type).toBe("file");
|
||||
expect(part.mediaType).toBe("application/pdf");
|
||||
expect(part.data.type).toBe("url");
|
||||
expect(part.data.url).toBeInstanceOf(URL);
|
||||
expect(part.data.url.toString()).toBe("https://example.com/a.pdf");
|
||||
});
|
||||
|
||||
it("round-trips a text-tagged file part", () => {
|
||||
const original = [
|
||||
{ role: "user", content: [{ type: "file", mediaType: "text/plain", data: { type: "text", text: "hello, file" }, filename: "a.txt" }] },
|
||||
];
|
||||
expect(openAIToVercel(vercelToOpenAI(original))).toEqual(original);
|
||||
});
|
||||
|
||||
it("preserves a provider reference (tagged and bare) instead of dropping it", () => {
|
||||
const tagged = [
|
||||
{ role: "user", content: [{ type: "file", mediaType: "application/pdf", data: { type: "reference", reference: { openai: "file-abc" } } }] },
|
||||
];
|
||||
expect(openAIToVercel(vercelToOpenAI(tagged))).toEqual(tagged);
|
||||
const bare = [{ role: "user", content: [{ type: "file", mediaType: "application/pdf", data: { openai: "file-abc" } }] }];
|
||||
expect(openAIToVercel(vercelToOpenAI(bare))).toEqual(bare);
|
||||
});
|
||||
});
|
||||
|
||||
describe("assistant text separators across formats", () => {
|
||||
it("openAIToAnthropic joins adjacent assistant text parts with a newline into one block", () => {
|
||||
const msgs = [
|
||||
{ role: "assistant", content: [{ type: "text", text: "first" }, { type: "text", text: "second" }] },
|
||||
] as OpenAIMessage[];
|
||||
expect(openAIToAnthropic(msgs)).toEqual([{ role: "assistant", content: "first\nsecond" }]);
|
||||
});
|
||||
|
||||
it("openAIToAnthropic keeps text parts separate when the array carries media", () => {
|
||||
const msgs = [
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "a" },
|
||||
{ type: "text", text: "b" },
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } },
|
||||
{ type: "text", text: "c" },
|
||||
],
|
||||
},
|
||||
] as OpenAIMessage[];
|
||||
expect(openAIToAnthropic(msgs)).toEqual([
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "a" },
|
||||
{ type: "text", text: "b" },
|
||||
{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw==" } },
|
||||
{ type: "text", text: "c" },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("round-trips Gemini and Anthropic assistant turns with two adjacent text parts before media exactly", () => {
|
||||
const gemini = [
|
||||
{ role: "model", parts: [{ text: "Hello " }, { text: "world" }, { inlineData: { mimeType: "image/png", data: "iVBORw==" } }] },
|
||||
];
|
||||
expect(openAIToGemini(geminiToOpenAI(gemini))).toEqual(gemini);
|
||||
const anthropic = [
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "Hello " },
|
||||
{ type: "text", text: "world" },
|
||||
{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw==" } },
|
||||
],
|
||||
},
|
||||
];
|
||||
expect(openAIToAnthropic(anthropicToOpenAI(anthropic))).toEqual(anthropic);
|
||||
});
|
||||
|
||||
it("openAIToGemini joins adjacent assistant text parts with a newline into one part", () => {
|
||||
const msgs = [
|
||||
{ role: "assistant", content: [{ type: "text", text: "first" }, { type: "text", text: "second" }] },
|
||||
] as OpenAIMessage[];
|
||||
expect(openAIToGemini(msgs)).toEqual([{ role: "model", parts: [{ text: "first\nsecond" }] }]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Anthropic user document blocks", () => {
|
||||
it("keeps a user document (PDF) beside its text as an image_url data URI", () => {
|
||||
const result = anthropicToOpenAI([
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "Summarize" },
|
||||
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: "JVBERi0=" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
expect(result).toEqual([
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "Summarize" },
|
||||
{ type: "image_url", image_url: { url: "data:application/pdf;base64,JVBERi0=" } },
|
||||
],
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps a document-only user turn instead of dropping it", () => {
|
||||
const result = anthropicToOpenAI([
|
||||
{ role: "user", content: [{ type: "document", source: { type: "url", url: "https://example.com/a.pdf" } }] },
|
||||
]);
|
||||
expect(result).toEqual([
|
||||
{ role: "user", content: [{ type: "image_url", image_url: { url: "https://example.com/a.pdf", mime_type: "application/pdf" } }] },
|
||||
]);
|
||||
});
|
||||
|
||||
it("round-trips a user text+document turn exactly through detection and conversion", () => {
|
||||
const original = [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "Summarize" },
|
||||
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: "JVBERi0=" } },
|
||||
],
|
||||
},
|
||||
];
|
||||
expect(detectFormat(original)).toBe("anthropic");
|
||||
expect(fromOpenAI(toOpenAI(original), "anthropic")).toEqual(original);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Anthropic document sources (text, content, file) and document fields", () => {
|
||||
it("keeps a document-only user turn with a text source, title and citations through the public path", () => {
|
||||
const input = [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{
|
||||
type: "document",
|
||||
source: { type: "text", media_type: "text/plain", data: "Critical contract: do not delete account 123" },
|
||||
title: "Contract",
|
||||
citations: { enabled: true },
|
||||
},
|
||||
],
|
||||
},
|
||||
];
|
||||
expect(detectFormat(input)).toBe("anthropic");
|
||||
const wire = toOpenAI(input);
|
||||
expect(wire).toHaveLength(1);
|
||||
expect((wire[0].content as any)[0].type).toBe("image_url");
|
||||
expect(fromOpenAI(wire, "anthropic")).toEqual(input);
|
||||
});
|
||||
|
||||
it("round-trips a mixed text + text-document turn with context exactly", () => {
|
||||
const input = [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "text", text: "Summarize" },
|
||||
{ type: "document", source: { type: "text", media_type: "text/plain", data: "clause 1\nclause 2" }, context: "Contract excerpt" },
|
||||
],
|
||||
},
|
||||
];
|
||||
expect(fromOpenAI(toOpenAI(input), "anthropic")).toEqual(input);
|
||||
});
|
||||
|
||||
it("carries content-source and file-source documents verbatim", () => {
|
||||
const input = [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{ type: "document", source: { type: "content", content: [{ type: "text", text: "clause 1" }] }, title: "Doc" },
|
||||
{ type: "document", source: { type: "file", file_id: "file_abc" } },
|
||||
],
|
||||
},
|
||||
];
|
||||
expect(fromOpenAI(toOpenAI(input), "anthropic")).toEqual(input);
|
||||
});
|
||||
|
||||
it("restores a PDF document's title, context and citations", () => {
|
||||
const input = [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
{
|
||||
type: "document",
|
||||
source: { type: "base64", media_type: "application/pdf", data: "JVBERi0=" },
|
||||
title: "Report",
|
||||
context: "Q3",
|
||||
citations: { enabled: true },
|
||||
},
|
||||
],
|
||||
},
|
||||
];
|
||||
expect(fromOpenAI(toOpenAI(input), "anthropic")).toEqual(input);
|
||||
});
|
||||
|
||||
it("round-trips an assistant document with a title", () => {
|
||||
const input = [
|
||||
{
|
||||
role: "assistant",
|
||||
content: [
|
||||
{ type: "text", text: "see attached" },
|
||||
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: "JVBERi0=" }, title: "Report" },
|
||||
],
|
||||
},
|
||||
];
|
||||
expect(fromOpenAI(toOpenAI(input), "anthropic")).toEqual(input);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user