fix(sdk): preserve Vercel AI SDK file and image parts through compression (#4048)

## Description

The TypeScript SDK's Vercel AI SDK converters lose or break media:

- **`headroomMiddleware()` / `withHeadroom(model)` drop every image and
document.** The provider prompt the middleware receives
(`params.prompt`) carries media only as `file` parts — in the installed
`@ai-sdk/provider` 4.0.19 a user message is
`Array<LanguageModelV3TextPart | LanguageModelV3FilePart>`; there is no
`image` part at that level. `vercelToOpenAI()` only handles `text` and
`image`, so a `file` part is silently removed and a file-only turn
becomes `content: []`. Because the middleware returns the original
params when compression did not happen, the media is lost exactly when
compression succeeds.
- **`compressVercelMessages()` crashes on binary images.** For
user-level `ModelMessage`s, an `image` part whose data is a `Uint8Array`
(or a bare base64 string) is stringified on the way out
(`"137,80,78,71"`) and `openAIToVercel()` then calls `new URL()` on it:
`TypeError: Invalid URL`, after a successful compression. `file` parts
(e.g. a PDF) are dropped on this path too.

Reproduced through the public `toOpenAI` / `fromOpenAI` on `main`
(`20ce054f`):

```text
prompt: text+file(image/png, bytes) | OpenAI: {"role":"user","content":"what is this?"}   | back: text only
prompt: file-only (application/pdf) | OpenAI: {"role":"user","content":[]}               | back: {"role":"user","content":[]}
message: image as Uint8Array        | OpenAI: image_url.url = "137,80,78,71"              | ROUND TRIP THROWS TypeError: Invalid URL
message: image as base64 string     | OpenAI: image_url.url = "iVBORw0KGgo="              | ROUND TRIP THROWS TypeError: Invalid URL
message: file part (pdf)            | OpenAI: {"role":"user","content":"summarise"}       | back: text only
```

Seven more, found in review (JerrettDavis and the Devin bot,
2026-10-09/14) and fixed here as well:

- **The public `compress()` could not even detect a media-only Vercel
conversation.** `detectFormat` knew Vercel only by `tool-call` /
`tool-result` parts, so a conversation whose only Vercel markers were
`file` / `image` parts was classified as OpenAI and sent as-is — a
`Uint8Array` serialises as `{"0":137,"1":80,…}` on the wire.
- **Assistant file parts were dropped by the explicit converters too.**
AI SDK 6's `AssistantContent` includes `FilePart` (generated media sent
back as history) and this package's peer range is `ai >=6.0.0`, but the
assistant branch of `vercelToOpenAI` kept only text and tool calls.
- **Assistant media was dropped again when converting back to Anthropic
or Gemini.** The first revision's `assistantText` helper kept only the
text of an assistant content array, so an assistant image or file in the
OpenAI shape (e.g. from a Vercel file part) vanished in `fromOpenAI(…,
"anthropic" | "gemini")` even though both targets have media parts.
- **AI SDK 7 / LanguageModelV4 file parts were dropped by the
middleware.** `wrapLanguageModel` in `ai` 7.x hands `transformParams` a
V4 prompt whose `file` parts carry `data` as a tagged union (`{ type:
"data" | "url" | "text" | "reference", … }`), not the bare
bytes/string/URL the converter read, so the attachment vanished before
compression. Provider references were silently removed too.
- **A conversation whose only Anthropic marker is a `document` block was
detected as OpenAI**, so `compress()` skipped the Anthropic converter —
and once detected, the Anthropic user branch dropped the document, since
it only knew images.
- **Adjacent assistant text parts lost their newline join** in the
Anthropic and Gemini reverse converters.
- **Anthropic document sources other than PDF were dropped, and document
fields were lost.** A document with a plain-text source (or a content /
file source) produced no media part — a document-only turn vanished —
and a PDF document came back without its `title`, `context` and
`citations`.

Same bug class as #3439 (Anthropic) and #3882 (Gemini), on the Vercel
adapter, fixed the same way: media becomes OpenAI `image_url` parts the
proxy passes through untouched, with the few facts the OpenAI shape
cannot hold carried on private keys so the reverse conversion rebuilds
the right Vercel part.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`sdk/typescript/src/utils/format.ts`**
- `vercelToOpenAI` user branch (forward): `image` **and** `file` parts
are converted. `Uint8Array` / `ArrayBuffer` / base64-string data →
`image_url` with a `data:<mediaType>;base64,…` URI (the media type
travels inside it; if the part has none, an empty media type is carried
rather than guessed); `URL` objects and URL strings → the URL, with the
part's media type on a private `mime_type` key (the same private key
#3882 introduced for Gemini, so one key serves every converter). A
`file` part additionally carries `part: "file"` and its `filename`.
Unconvertible data (e.g. provider file references) is skipped without
deleting the turn. A turn with media is emitted as an ordered part
array; **text-only turns are unchanged** (still joined into a string).
- **LanguageModelV4 / AI SDK 7 tagged data:** the carrier records the
tag on a private `data_tag` key — `data` and `text` travel as data: URIs
and are restored tagged (base64 inside `data`, decoded text inside
`text`), `url` is restored as a `URL` object inside the tag, and a
`reference` (tagged, or a bare AI SDK 6 `ProviderReference`) is carried
verbatim on a `reference` key with an empty data: URI as the opaque url
and returned untouched instead of being removed. Bare AI SDK 6 inputs
restore bare, so standalone `ModelMessage` handling is unchanged.
- `openAIToVercel` user branch (reverse): a data URI → base64 data plus
its media type; an http(s) URL → a `URL` object (unchanged behaviour,
existing test kept); anything else → left as a string. **Never throws.**
Emits `{ type: "file", mediaType, data, filename? }` when `part ===
"file"` or the media type is not `image/*`, else `{ type: "image",
image, mediaType? }`. Base64 strings are valid data for both the
provider prompt (`LanguageModelV3DataContent = Uint8Array | string |
URL`, string = base64) and `ModelMessage` (`DataContent` accepts
base64), so byte inputs come back as base64 strings — equal content, not
the same object.
- `detectFormat`: a `file` part carrying `data` / `mediaType` (OpenAI's
`file` part carries a `file` object instead) or an `image` part carrying
`image` (Anthropic's carries `source`) now marks the Vercel format; an
Anthropic `document` block with a `source.type` marks Anthropic. OpenAI
`image_url` parts and Anthropic `image` blocks are unaffected, with
tests for both.
- `vercelToOpenAI` assistant branch: a turn with `file` parts becomes an
ordered part array (text + `image_url` with the carriers above);
text-only assistant turns still convert to a string. `openAIToVercel`
rebuilds text and `file` parts with the assistant role.
- `openAIToAnthropic` / `openAIToGemini` assistant branches: a text-only
array joins with `"\n"` into one block/part, as the forward converters
join text-only turns; an array carrying media keeps every part separate,
because the forward converters keep exact parts in that case and a round
trip must not add or move characters (`groupAssistantParts`); assistant
media in the OpenAI shape is rebuilt in order relative to text —
Anthropic `image/*` → image block, `application/pdf` → document block
(base64 or url source, per the SDK's `DocumentBlockParam`); Gemini data
URIs → `inlineData`, URL-backed files → `fileData` with the carried MIME
type. Media Anthropic has no block for is omitted rather than mis-typed.
The Anthropic user branch now maps PDFs to document blocks as well.
- `anthropicToOpenAI` user branch: `document` blocks are kept beside
images — base64 PDF → data: URI; url source → url with `mime_type`;
plain-text source → `data:text/plain;base64,…` tagged `source_type:
"text"` and restored as text; content and file sources carried verbatim
on a `source` key (preserved, not removed); `title` / `context` /
`citations` / `cache_control` carried on a `document` key and
re-attached — so every `DocumentBlockParam` round-trips exactly and a
document-only turn is no longer dropped.
- `anthropicToOpenAI` assistant and `geminiToOpenAI` model branches:
image/document blocks and `inlineData`/`fileData` become ordered parts,
so both formats round-trip exactly (Gemini image-generation output is a
real case).
- Helpers: `bytesToBase64` / `base64ToText` (use `Buffer` when present,
`btoa`/`atob` otherwise), `parseUrl`, `isProviderReference`,
`vercelMediaImageUrl`, `vercelMediaPart`, `groupAssistantParts`,
`anthropicMediaImageUrl` / `anthropicMediaBlock` (with the
`AnthropicMediaImageUrl` carrier and `ANTHROPIC_DOCUMENT_FIELDS`).
- **`sdk/typescript/src/types.ts`** — `AssistantMessage.content` widens
from `string | null` to `string | ContentPart[] | null` so an assistant
turn can carry media parts; OpenAI's own API already accepts an array of
parts as assistant content. Additive for producers; the only readers in
the SDK (`hooks.ts`) already branch on arrays, and the
`plugins/openclaw` converter defines its own message type. The public
type test is updated to match.
- **`sdk/typescript/test/utils/detect-format.test.ts`** — media-only
Vercel turns (file part; image part) detect as `vercel`; a document-only
Anthropic turn detects as `anthropic`; OpenAI `image_url` and Anthropic
`image` detection unchanged.
- **`sdk/typescript/test/compress.test.ts`** — the public `compress()`
with a media-only Vercel turn and a mocked proxy: the wire body carries
an `image_url` data URI (and no `"0":137`), and the result comes back as
a Vercel `file` part; a document-only Anthropic turn through
`compress()` against a pass-through mock comes back as the same document
block.
- **`sdk/typescript/test/utils/format.test.ts`** — Anthropic document
sources (document-only text turn through detection and conversion, mixed
text+document with context, content and file sources verbatim, PDF
fields, assistant document); V4 tagged data (data, url, text, reference,
and a bare reference) forward/reverse/round trip; assistant text
separators (text-only join; media-bearing arrays keep parts; exact
Gemini and Anthropic round trips with adjacent text before media);
Anthropic user documents (text+document, document-only, round trip
through detection); cross-format assistant media: the review thread's
Vercel → OpenAI → Gemini / Anthropic example, both reverse converters
(image; PDF base64 and url; text-only backward compat), both forward
converters, exact Anthropic and Gemini assistant round trips; Vercel
assistant: text + file forward, reverse with the assistant role, round
trip, text-only backward compat; user: forward (prompt `file` image
bytes with text; file-only PDF with filename; `Uint8Array` image;
bare-base64 image; URL-backed file with carried media type), reverse
(data URI marked `file`; image data URI; URL-backed file; unparseable
url does not throw), and exact round trips (prompt text+file; byte
image).
- **`sdk/typescript/test/adapters/vercel-ai.test.ts`** —
`headroomMiddleware`: a prompt `file` part reaches `/v1/compress` as an
`image_url` data URI and comes back in the returned prompt as a `file`
part; the same for an assistant file turn in the prompt; a public
`transformParams` round trip typed with the actual
`LanguageModelV4FilePart` / `LanguageModelV4Prompt` (tagged data
restored tagged) and a provider reference passed through untouched;
`compressVercelMessages` no longer throws on byte image data.

## Testing

- [x] Unit tests pass (`npm test` — vitest, incl. typecheck; also run by
the new `sdk-typescript.yml` job)
- [x] Type checking passes (`npm run typecheck` — tsc --noEmit)
- [x] Build passes (`npm run build` — tsup)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cd sdk/typescript && npx vitest run test/utils/format.test.ts test/utils/detect-format.test.ts test/compress.test.ts \
    test/adapters/vercel-ai.test.ts test/adapters/anthropic.test.ts test/adapters/gemini.test.ts
 Test Files  6 passed (6)
      Tests  147 passed (147)

$ npm test                       # full suite
 Test Files  19 passed | 5 skipped (24)
      Tests  404 passed | 33 skipped (437)
Type Errors  no errors

$ npm run typecheck              # tsc --noEmit
(clean, no errors)

$ npm run build                  # tsup
CJS / ESM / DTS Build success
```

## Real Behavior Proof

- Environment: macOS (Apple silicon), Node v22.16.0, npm 10.9.2; `cd
sdk/typescript && npm ci` (`ai` 7.0.119, `@ai-sdk/provider` 4.0.19,
`@ai-sdk/provider-utils` 5.0.50).
- Exact command / steps: (1) the block in the description — a `node
--input-type=module` script calling the built `toOpenAI` / `fromOpenAI`
from `dist/index.js` on `main`; (2) the thirteen new tests run on `main`
before the change and on this branch after; (3) for the review findings:
`detectFormat` + `toOpenAI` on a media-only Vercel turn, and the
converters on an assistant text+file turn, both against the first
revision; then the eight new tests before and after the second commit;
(4) second review round: `fromOpenAI(vercelToOpenAI(<assistant text +
file>), "gemini")` on the second revision, then the eleven new tests
before and after the third commit; then `npm test`, `npm run typecheck`,
`npm run build`.
- Observed result: on `main` all thirteen fail for the expected reasons
— media parts missing from the converted output, and `AssertionError:
expected [Function] to not throw an error but 'TypeError: Invalid URL'
was thrown` for the byte/base64 cases (the middleware test fails the
same way). With the first commit all 71 targeted tests pass. Review
findings reproduced on that revision: `detect = openai` and the wire
payload
`{"type":"file","mediaType":"image/png","data":{"0":137,"1":80,"2":78,"3":71}}`
for a media-only turn; `{role:"assistant",content:"generated image"}`
for an assistant text+file turn (file gone). The eight new tests fail on
that revision (seven runtime, one type assertion) and pass after the
second commit. Second round: on that revision the Gemini conversion of a
Vercel assistant file returned `{ role: "model", parts: [{ text:
"generated image" }] }` (media gone); the eleven new tests fail there
and pass after the third commit: 127 targeted tests, full suite 384
passed | 33 skipped, typecheck and build clean.
- **Live proxy round trip** (replaces the earlier "not tested"):
`headroom proxy` 0.40.0 from the published wheel (`uvx --from
'headroom-ai[proxy]' headroom proxy --port 8799`), the built middleware
from this branch, a V4 prompt with a 120-line service log, a data-tagged
PNG, a provider-reference PDF and an assistant file turn; then the
public `compress()` with a media-only Vercel turn. Captured on
`b00b388b`; the later `f5dbdbcc` changes only the Anthropic user branch
and the text-part rule, not the Vercel path exercised here. Output (text
parts shown as `text(<length>)`):

```text
proxy http://127.0.0.1:8799/v1/compress status 200 | tokens_before 4497 tokens_after 532 tokens_saved 3965 | transforms ["router:text_block:log"]
SENT     : [["text(10758)","{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64,iVBORw==\",\"part\":\"file\",\"data_tag\":\"data\"}}","{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:application/pdf;base64,\",\"part\":\"file\",\"data_tag\":\"reference\",\"reference\":{\"openai\":\"file-abc\"}}}"],["text(15)","{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64,iVBORw==\",\"part\":\"file\",\"data_tag\":\"data\"}}"]]
RETURNED : [["text(29)","{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64,iVBORw==\",\"part\":\"file\",\"data_tag\":\"data\"}}","{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:application/pdf;base64,\",\"part\":\"file\",\"data_tag\":\"reference\",\"reference\":{\"openai\":\"file-abc\"}}}"],["text(15)","{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64,iVBORw==\",\"part\":\"file\",\"data_tag\":\"data\"}}"]]
RESTORED : [["text(29)","{\"type\":\"file\",\"mediaType\":\"image/png\",\"data\":{\"type\":\"data\",\"data\":\"iVBORw==\"}}","{\"type\":\"file\",\"mediaType\":\"application/pdf\",\"data\":{\"type\":\"reference\",\"reference\":{\"openai\":\"file-abc\"}}}"],["text(15)","{\"type\":\"file\",\"mediaType\":\"image/png\",\"data\":{\"type\":\"data\",\"data\":\"iVBORw==\"}}"]]
middleware returned original params (no compression)? false
compress(): sent part[1] = {"type":"image_url","image_url":{"url":"data:image/png;base64,iVBORw==","part":"file"}} | returned part[1] = {"type":"image_url","image_url":{"url":"data:image/png;base64,iVBORw==","part":"file"}} | result part[1] = {"type":"file","mediaType":"image/png","data":"iVBORw=="} | compressed = true saved 3950
```

The proxy compressed the log, returned every media part with the private
carrier keys intact, and the restored prompt carries the tagged V4 parts
and the reference unchanged; the public `compress()` path detected
Vercel, compressed, and restored the file part.
- Not tested: a live provider call downstream of the proxy.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. Client-side TypeScript SDK
conversion fix; no server/runtime feature flag or rollout channel
involved.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: Only the buggy paths — media that was
dropped is preserved, inputs that threw now convert, and media-only
Vercel conversations are now detected as Vercel. Text-only turns, tool
calls/results and the http-URL image case are byte-identical to before.
The one public type change is additive: `AssistantMessage.content` also
admits an array of content parts.
- Kill switch / disable path: N/A — correctness fix, not a gated
feature.
- Unsafe override required: No.
- Qualification impact: None — additive media handling; existing
conversion tests unchanged and green.
- Rollback path: Revert this PR; no persisted state, migrations, or
config to unwind.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Out of scope, left as-is and pre-existing: assistant `reasoning` parts
(dropped by the assistant branch of `vercelToOpenAI`); media inside
`tool-result` parts; `providerOptions` on parts. Each is a separate
follow-up.
- Anthropic has blocks only for images and PDFs (and text documents);
other media reaching `openAIToAnthropic` is omitted rather than emitted
as a mis-typed block.
- Revisions after review: detection of media-only Vercel conversations
and assistant file parts (second commit), including the
`AssistantMessage.content` widening described above; assistant media
through the Anthropic and Gemini converters in both directions, the
`mime_type` carrier rename and the removal of `assistantText` (third
commit); LanguageModelV4 tagged data and provider references, Anthropic
document detection, assistant text separators and the live proxy proof
(fourth commit); user documents kept in the Anthropic user branch and
exact text parts beside media (fifth commit); Anthropic text / content /
file document sources and document fields (sixth commit, on top of the
maintainer's `main` sync).
- Large documents travel as base64 inside the request body, as
`inlineData` already does for Gemini (#3882); the proxy counts image
parts at a fixed cost.
- CHANGELOG left untouched — the `no-manual-changelog` CI guard owns it;
the Conventional Commit title drives the entry.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Shubham Srivastava
2026-10-24 08:39:09 -07:00
committed by GitHub
co-authored by JerrettDavis
parent 125a703a65
commit 1668a6c0dd
7 changed files with 1233 additions and 30 deletions
+5 -1
View File
@@ -38,7 +38,11 @@ export interface UserMessage {
export interface AssistantMessage {
role: "assistant";
content: string | null;
/**
* A string, or an ordered part array when the assistant turn carries media
* (e.g. a Vercel AI SDK file part): text parts plus `image_url` parts.
*/
content: string | ContentPart[] | null;
tool_calls?: ToolCall[];
}
+324 -26
View File
@@ -14,6 +14,7 @@
import type {
OpenAIMessage,
AssistantMessage,
ContentPart,
ToolCall,
} from "../types.js";
@@ -51,8 +52,14 @@ export function detectFormat(messages: any[]): MessageFormat {
if (part.type === "tool-call" || part.type === "tool-result") return "vercel";
// Anthropic: underscored types
if (part.type === "tool_use" || part.type === "tool_result") return "anthropic";
// Anthropic: image with source.type
if (part.type === "image" && part.source?.type) return "anthropic";
// Anthropic: image / document with source.type
if ((part.type === "image" || part.type === "document") && part.source?.type) return "anthropic";
// Vercel: `file` parts carry data/mediaType (OpenAI's `file` part carries a `file` object);
// `image` parts carry `image` (Anthropic's carry `source`)
if (part && typeof part === "object") {
if (part.type === "file" && ("data" in part || "mediaType" in part)) return "vercel";
if (part.type === "image" && "image" in part) return "vercel";
}
}
}
}
@@ -88,6 +95,74 @@ function openAIImageBlock(url: string): any {
return { type: "image", source: { type: "url", url } };
}
type AnthropicMediaImageUrl = {
url: string;
mime_type?: string;
// A plain-text document source travels as a data: URI and is tagged so it is restored as
// text, not base64; sources the OpenAI shape cannot express (content, file) are carried
// verbatim on `source`. The document's own fields are carried on `document`.
source_type?: "text";
source?: any;
document?: Record<string, any>;
};
const ANTHROPIC_DOCUMENT_FIELDS = ["title", "context", "citations", "cache_control"] as const;
// Anthropic media block -> OpenAI image_url object, or null if unconvertible: image blocks via
// anthropicImageUrl; document blocks by source — base64 PDF as a data: URI, url with the MIME
// type carried on the private `mime_type` key, plain text as a tagged data: URI, and content /
// file sources carried verbatim — plus the document's title / context / citations / cache_control.
function anthropicMediaImageUrl(block: any): AnthropicMediaImageUrl | null {
if (block.type === "image") {
const url = anthropicImageUrl(block.source);
return url ? { url } : null;
}
if (block.type !== "document" || !block.source || typeof block.source !== "object") return null;
const src = block.source;
let out: AnthropicMediaImageUrl;
if (src.type === "base64" && src.media_type && src.data) {
out = { url: `data:${src.media_type};base64,${src.data}` };
} else if (src.type === "url" && typeof src.url === "string") {
out = { url: src.url, mime_type: "application/pdf" };
} else if (src.type === "text" && typeof src.data === "string") {
out = {
url: `data:${src.media_type || "text/plain"};base64,${bytesToBase64(new TextEncoder().encode(src.data))}`,
source_type: "text",
};
} else {
out = { url: "data:;base64,", source: src };
}
const fields: Record<string, any> = {};
for (const k of ANTHROPIC_DOCUMENT_FIELDS) if (block[k] !== undefined) fields[k] = block[k];
if (Object.keys(fields).length > 0) out.document = fields;
return out;
}
// OpenAI image_url -> Anthropic media block, or null when Anthropic has no block for that media:
// a carried document source or text document is rebuilt as a document block with its fields;
// application/pdf -> document block (base64 or url source); image/* (or unknown) -> image block.
function anthropicMediaBlock(imageUrl: any): any {
const url: string = imageUrl.url;
const m = /^data:([^;,]*);base64,(.*)$/s.exec(url);
const mediaType: string | undefined =
(m && m[1]) || (typeof imageUrl.mime_type === "string" ? imageUrl.mime_type : undefined);
const fields = imageUrl.document && typeof imageUrl.document === "object" ? imageUrl.document : {};
if (imageUrl.source && typeof imageUrl.source === "object") {
return { type: "document", source: imageUrl.source, ...fields };
}
if (imageUrl.source_type === "text" && m) {
return { type: "document", source: { type: "text", media_type: "text/plain", data: base64ToText(m[2]) }, ...fields };
}
if (mediaType === "application/pdf") {
const source = m
? { type: "base64", media_type: "application/pdf", data: m[2] }
: { type: "url", url };
return { type: "document", source, ...fields };
}
if (mediaType === undefined || mediaType.startsWith("image/")) return openAIImageBlock(url);
return null;
}
export function anthropicToOpenAI(messages: any[]): OpenAIMessage[] {
const result: OpenAIMessage[] = [];
@@ -99,15 +174,15 @@ export function anthropicToOpenAI(messages: any[]): OpenAIMessage[] {
}
if (Array.isArray(msg.content)) {
const toolResults = msg.content.filter((b: any) => b.type === "tool_result");
const hasImages = msg.content.some((b: any) => b.type === "image");
const hasMedia = msg.content.some((b: any) => b.type === "image" || b.type === "document");
if (hasImages) {
if (hasMedia) {
const parts = msg.content
.filter((b: any) => b.type === "text" || b.type === "image")
.filter((b: any) => b.type === "text" || b.type === "image" || b.type === "document")
.map((b: any) => {
if (b.type === "text") return { type: "text" as const, text: b.text };
const url = anthropicImageUrl(b.source);
return url ? { type: "image_url" as const, image_url: { url } } : null;
const imageUrl = anthropicMediaImageUrl(b);
return imageUrl ? { type: "image_url" as const, image_url: imageUrl } : null;
})
.filter((p: any): p is NonNullable<typeof p> => p !== null);
@@ -147,10 +222,22 @@ export function anthropicToOpenAI(messages: any[]): OpenAIMessage[] {
if (Array.isArray(msg.content)) {
const textBlocks = msg.content.filter((b: any) => b.type === "text");
const toolUseBlocks = msg.content.filter((b: any) => b.type === "tool_use");
const hasMedia = msg.content.some((b: any) => b.type === "image" || b.type === "document");
const content = textBlocks.length > 0
? textBlocks.map((b: any) => b.text).join("\n")
: null;
let content: AssistantMessage["content"];
if (hasMedia) {
// Keep generated media as ordered parts (see AssistantMessage.content).
content = msg.content
.filter((b: any) => b.type === "text" || b.type === "image" || b.type === "document")
.map((b: any) => {
if (b.type === "text") return { type: "text" as const, text: b.text };
const imageUrl = anthropicMediaImageUrl(b);
return imageUrl ? { type: "image_url" as const, image_url: imageUrl } : null;
})
.filter((p: any): p is NonNullable<typeof p> => p !== null);
} else {
content = textBlocks.length > 0 ? textBlocks.map((b: any) => b.text).join("\n") : null;
}
const openaiMsg: AssistantMessage = { role: "assistant", content };
if (toolUseBlocks.length > 0) {
@@ -190,7 +277,7 @@ export function openAIToAnthropic(messages: OpenAIMessage[]): any[] {
role: "user",
content: msg.content.map((p) => {
if (p.type === "text") return { type: "text", text: p.text };
if (p.type === "image_url") return openAIImageBlock(p.image_url.url);
if (p.type === "image_url") return anthropicMediaBlock(p.image_url) ?? { type: "text", text: "" };
return { type: "text", text: "" };
}),
});
@@ -200,7 +287,17 @@ export function openAIToAnthropic(messages: OpenAIMessage[]): any[] {
if (msg.role === "assistant") {
const blocks: any[] = [];
if (msg.content) blocks.push({ type: "text", text: msg.content });
if (typeof msg.content === "string") {
if (msg.content) blocks.push({ type: "text", text: msg.content });
} else if (Array.isArray(msg.content)) {
for (const g of groupAssistantParts(msg.content)) {
if ("text" in g) blocks.push({ type: "text", text: g.text });
else {
const block = anthropicMediaBlock(g.image_url);
if (block) blocks.push(block);
}
}
}
if (msg.tool_calls) {
for (const tc of msg.tool_calls) {
blocks.push({
@@ -236,6 +333,167 @@ export function openAIToAnthropic(messages: OpenAIMessage[]): any[] {
// Vercel AI SDK → OpenAI
// ============================================================
function bytesToBase64(bytes: Uint8Array): string {
if (typeof Buffer !== "undefined") return Buffer.from(bytes).toString("base64");
let binary = "";
for (let i = 0; i < bytes.length; i += 0x8000) {
binary += String.fromCharCode(...bytes.subarray(i, i + 0x8000));
}
return btoa(binary);
}
function parseUrl(value: string): URL | null {
try {
return new URL(value);
} catch {
return null;
}
}
type VercelMediaImageUrl = {
url: string;
mime_type?: string;
part?: "file";
filename?: string;
// AI SDK 7 / LanguageModelV4 tagged file data: the tag the input used, so the reverse
// conversion restores the same shape; absent for the bare AI SDK 6 forms.
data_tag?: "data" | "url" | "text" | "reference";
// A provider-managed file reference, carried verbatim (it has no bytes or url to convert).
reference?: Record<string, string>;
};
function base64ToText(b64: string): string {
if (typeof Buffer !== "undefined") return Buffer.from(b64, "base64").toString("utf8");
const binary = atob(b64);
const bytes = new Uint8Array(binary.length);
for (let i = 0; i < binary.length; i++) bytes[i] = binary.charCodeAt(i);
return new TextDecoder().decode(bytes);
}
// A bare AI SDK 6 provider reference: a plain string-valued record with no `type`.
function isProviderReference(v: any): boolean {
return (
!!v &&
typeof v === "object" &&
!Array.isArray(v) &&
!(v instanceof Uint8Array) &&
!(v instanceof ArrayBuffer) &&
!(v instanceof URL) &&
v.type === undefined &&
Object.values(v).every((x) => typeof x === "string")
);
}
// Vercel `image` / `file` part -> OpenAI image_url object, or null if the data is unconvertible.
// Accepts the bare AI SDK 6 forms (bytes, base64, data URL, URL, provider reference) and the
// AI SDK 7 / LanguageModelV4 tagged forms ({ type: "data" | "url" | "text" | "reference" }).
// Bytes, base64 and text travel as a data: URI, which carries the media type; URLs stay URLs
// with the media type on a private `mime_type` key; references are carried verbatim on
// `reference`. `part: "file"`, `filename` and `data_tag` are carried so the reverse conversion
// rebuilds the right part in the same shape. The proxy passes non-text parts through unchanged.
function vercelMediaImageUrl(p: any): VercelMediaImageUrl | null {
let data = p.type === "file" ? p.data : p.image;
const mediaType: string | undefined = typeof p.mediaType === "string" ? p.mediaType : p.mimeType;
const out: VercelMediaImageUrl = { url: "" };
if (p.type === "file") out.part = "file";
if (typeof p.filename === "string") out.filename = p.filename;
if (data && typeof data === "object" && typeof data.type === "string") {
if (data.type === "data") {
out.data_tag = "data";
data = data.data;
} else if (data.type === "url") {
out.data_tag = "url";
data = data.url;
} else if (data.type === "text" && typeof data.text === "string") {
out.data_tag = "text";
out.url = `data:${mediaType ?? "text/plain"};base64,${bytesToBase64(new TextEncoder().encode(data.text))}`;
return out;
} else if (data.type === "reference" && data.reference && typeof data.reference === "object") {
out.data_tag = "reference";
out.reference = data.reference;
out.url = `data:${mediaType ?? ""};base64,`;
return out;
} else {
return null;
}
} else if (isProviderReference(data)) {
out.reference = data;
out.url = `data:${mediaType ?? ""};base64,`;
return out;
}
let carryType = false;
if (data instanceof Uint8Array || data instanceof ArrayBuffer) {
const bytes = data instanceof Uint8Array ? data : new Uint8Array(data);
out.url = `data:${mediaType ?? ""};base64,${bytesToBase64(bytes)}`;
} else if (data instanceof URL) {
out.url = data.toString();
carryType = true;
} else if (typeof data === "string") {
if (data.startsWith("data:")) out.url = data;
else if (parseUrl(data)) {
out.url = data;
carryType = true;
} else out.url = `data:${mediaType ?? ""};base64,${data}`;
} else {
return null;
}
if (carryType && mediaType) out.mime_type = mediaType;
return out;
}
// OpenAI image_url -> Vercel `image` or `file` part, in the shape the forward conversion saw
// (bare, or LanguageModelV4-tagged when `data_tag` is carried). A data: URI yields base64 data
// plus its media type; an http(s) url yields a URL object; anything else stays a string; a
// carried provider reference comes back verbatim. Never throws.
function vercelMediaPart(imageUrl: any): any {
const url: string = imageUrl.url;
const tag: string | undefined = typeof imageUrl.data_tag === "string" ? imageUrl.data_tag : undefined;
const m = /^data:([^;,]*);base64,(.*)$/s.exec(url);
let mediaType: string | undefined = typeof imageUrl.mime_type === "string" ? imageUrl.mime_type : undefined;
if (m && m[1]) mediaType = m[1];
let data: any;
if (imageUrl.reference && typeof imageUrl.reference === "object") {
data = tag === "reference" ? { type: "reference", reference: imageUrl.reference } : imageUrl.reference;
} else if (m) {
data = m[2];
if (tag === "data") data = { type: "data", data: m[2] };
else if (tag === "text") data = { type: "text", text: base64ToText(m[2]) };
} else {
const parsed = parseUrl(url);
data = parsed ?? url;
if (tag === "url") data = { type: "url", url: parsed ?? url };
}
const isFile = imageUrl.part === "file" || (mediaType !== undefined && !mediaType.startsWith("image/"));
if (isFile) {
const part: any = { type: "file", mediaType: mediaType ?? "application/octet-stream", data };
if (typeof imageUrl.filename === "string") part.filename = imageUrl.filename;
return part;
}
const part: any = { type: "image", image: data };
if (mediaType) part.mediaType = mediaType;
return part;
}
// Assistant content parts for the Anthropic / Gemini reverse converters. A text-only array
// collapses to one text joined with "\n" (the forward converters join text-only turns the same
// way); an array carrying media keeps every part separate, because the forward converters keep
// the exact parts in that case and a round trip must not add or move characters.
function groupAssistantParts(parts: ContentPart[]): Array<{ text: string } | { image_url: any }> {
const hasMedia = parts.some((p) => p.type === "image_url");
if (!hasMedia) {
const text = parts.filter((p) => p.type === "text").map((p) => (p as any).text).join("\n");
return text ? [{ text }] : [];
}
const out: Array<{ text: string } | { image_url: any }> = [];
for (const p of parts) {
if (p.type === "text") out.push({ text: p.text });
else if (p.type === "image_url") out.push({ image_url: p.image_url });
}
return out;
}
export function vercelToOpenAI(messages: any[]): OpenAIMessage[] {
const result: OpenAIMessage[] = [];
@@ -252,21 +510,19 @@ export function vercelToOpenAI(messages: any[]): OpenAIMessage[] {
}
const parts = Array.isArray(msg.content) ? msg.content : [];
const textParts = parts.filter((p: any) => p.type === "text");
const imageParts = parts.filter((p: any) => p.type === "image");
const mediaParts = parts.filter((p: any) => p.type === "image" || p.type === "file");
if (imageParts.length === 0 && textParts.length > 0) {
if (mediaParts.length === 0 && textParts.length > 0) {
result.push({ role: "user", content: textParts.map((p: any) => p.text).join("") });
} else {
const openaiParts = parts
.filter((p: any) => p.type === "text" || p.type === "image")
.filter((p: any) => p.type === "text" || p.type === "image" || p.type === "file")
.map((p: any) => {
if (p.type === "text") return { type: "text" as const, text: p.text };
if (p.type === "image") {
const url = p.image instanceof URL ? p.image.toString() : String(p.image);
return { type: "image_url" as const, image_url: { url } };
}
return { type: "text" as const, text: "" };
});
const imageUrl = vercelMediaImageUrl(p);
return imageUrl ? { type: "image_url" as const, image_url: imageUrl } : null;
})
.filter((p: any): p is NonNullable<typeof p> => p !== null);
result.push({ role: "user", content: openaiParts });
}
continue;
@@ -280,8 +536,22 @@ export function vercelToOpenAI(messages: any[]): OpenAIMessage[] {
const parts = Array.isArray(msg.content) ? msg.content : [];
const textParts = parts.filter((p: any) => p.type === "text");
const toolCallParts = parts.filter((p: any) => p.type === "tool-call");
const hasFiles = parts.some((p: any) => p.type === "file");
const content = textParts.length > 0 ? textParts.map((p: any) => p.text).join("") : null;
let content: AssistantMessage["content"];
if (hasFiles) {
// AI SDK assistant content may carry generated files; keep them as ordered parts.
content = parts
.filter((p: any) => p.type === "text" || p.type === "file")
.map((p: any) => {
if (p.type === "text") return { type: "text" as const, text: p.text };
const imageUrl = vercelMediaImageUrl(p);
return imageUrl ? { type: "image_url" as const, image_url: imageUrl } : null;
})
.filter((p: any): p is NonNullable<typeof p> => p !== null);
} else {
content = textParts.length > 0 ? textParts.map((p: any) => p.text).join("") : null;
}
const openaiMsg: AssistantMessage = { role: "assistant", content };
if (toolCallParts.length > 0) {
@@ -343,7 +613,7 @@ export function openAIToVercel(messages: OpenAIMessage[]): any[] {
} else if (Array.isArray(msg.content)) {
const parts = msg.content.map((p) => {
if (p.type === "text") return { type: "text", text: p.text };
if (p.type === "image_url") return { type: "image", image: new URL(p.image_url.url) };
if (p.type === "image_url") return vercelMediaPart(p.image_url);
return { type: "text", text: "" };
});
result.push({ role: "user", content: parts });
@@ -353,7 +623,14 @@ export function openAIToVercel(messages: OpenAIMessage[]): any[] {
if (msg.role === "assistant") {
const parts: any[] = [];
if (msg.content) parts.push({ type: "text", text: msg.content });
if (typeof msg.content === "string") {
if (msg.content) parts.push({ type: "text", text: msg.content });
} else if (Array.isArray(msg.content)) {
for (const p of msg.content) {
if (p.type === "text") parts.push({ type: "text", text: p.text });
else if (p.type === "image_url") parts.push(vercelMediaPart(p.image_url));
}
}
if (msg.tool_calls) {
for (const tc of msg.tool_calls) {
toolNames.set(tc.id, tc.function.name);
@@ -480,8 +757,22 @@ export function geminiToOpenAI(messages: any[]): OpenAIMessage[] {
if (role === "assistant") {
const textParts = parts.filter((p: any) => p.text !== undefined);
const funcCalls = parts.filter((p: any) => p.functionCall);
const hasMedia = parts.some((p: any) => p.inlineData || p.fileData);
const content = textParts.length > 0 ? textParts.map((p: any) => p.text).join("\n") : null;
let content: AssistantMessage["content"];
if (hasMedia) {
// Keep generated media (e.g. image output) as ordered parts.
content = parts
.filter((p: any) => p.text !== undefined || p.inlineData || p.fileData)
.map((p: any) => {
if (p.text !== undefined) return { type: "text" as const, text: p.text };
const imageUrl = geminiMediaImageUrl(p);
return imageUrl ? { type: "image_url" as const, image_url: imageUrl } : null;
})
.filter((p: any): p is NonNullable<typeof p> => p !== null);
} else {
content = textParts.length > 0 ? textParts.map((p: any) => p.text).join("\n") : null;
}
const openaiMsg: AssistantMessage = { role: "assistant", content };
if (funcCalls.length > 0) {
@@ -533,7 +824,14 @@ export function openAIToGemini(messages: OpenAIMessage[]): any[] {
if (msg.role === "assistant") {
const parts: any[] = [];
if (msg.content) parts.push({ text: msg.content });
if (typeof msg.content === "string") {
if (msg.content) parts.push({ text: msg.content });
} else if (Array.isArray(msg.content)) {
for (const g of groupAssistantParts(msg.content)) {
if ("text" in g) parts.push({ text: g.text });
else parts.push(geminiMediaPart(g.image_url));
}
}
if (msg.tool_calls) {
for (const tc of msg.tool_calls) {
parts.push({
+160 -1
View File
@@ -1,5 +1,5 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import type { LanguageModelV3 } from "@ai-sdk/provider";
import type { LanguageModelV3, LanguageModelV4FilePart, LanguageModelV4Prompt } from "@ai-sdk/provider";
import {
headroomMiddleware,
compressVercelMessages,
@@ -134,6 +134,149 @@ describe("headroomMiddleware", () => {
const body = JSON.parse(mockFetch.mock.calls[0][1].body);
expect(body.model).toBe("claude-sonnet-4-5-20250929");
});
it("keeps a provider-prompt file part through compression", async () => {
// The proxy only compresses text and passes the media part back untouched.
mockFetch.mockResolvedValueOnce(
mockCompressResponse([
{
role: "user",
content: [
{ type: "text", text: "compressed" },
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file" } },
],
},
]),
);
const middleware = headroomMiddleware({ baseUrl: "http://localhost:8787" });
const out = await middleware.transformParams({
params: {
prompt: [
{
role: "user",
content: [
{ type: "text", text: "what is this?" },
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]) },
],
},
],
},
model: {},
type: "generate",
});
const body = JSON.parse(mockFetch.mock.calls[0][1].body);
expect(body.messages[0].content[1]).toEqual({
type: "image_url",
image_url: { url: "data:image/png;base64,iVBORw==", part: "file" },
});
expect(out.prompt[0].content[1]).toEqual({ type: "file", mediaType: "image/png", data: "iVBORw==" });
});
it("keeps an assistant file part (generated media) through compression", async () => {
mockFetch.mockResolvedValueOnce(
mockCompressResponse([
{ role: "user", content: "draw a cat" },
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file" } },
],
},
]),
);
const middleware = headroomMiddleware({ baseUrl: "http://localhost:8787" });
const out = await middleware.transformParams({
params: {
prompt: [
{ role: "user", content: [{ type: "text", text: "draw a cat" }] },
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]) },
],
},
],
},
model: {},
type: "generate",
});
const body = JSON.parse(mockFetch.mock.calls[0][1].body);
expect(body.messages[1]).toEqual({
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file" } },
],
});
expect(out.prompt[1]).toEqual({
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "file", mediaType: "image/png", data: "iVBORw==" },
],
});
});
it("round-trips a LanguageModelV4 file part (tagged data) through transformParams", async () => {
const filePart: LanguageModelV4FilePart = {
type: "file",
mediaType: "image/png",
data: { type: "data", data: new Uint8Array([137, 80, 78, 71]) },
};
const prompt: LanguageModelV4Prompt = [{ role: "user", content: [{ type: "text", text: "what is this?" }, filePart] }];
// The proxy keeps the media part and compresses only the text.
mockFetch.mockResolvedValueOnce(
mockCompressResponse([
{
role: "user",
content: [
{ type: "text", text: "compressed" },
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file", data_tag: "data" } },
],
},
]),
);
const middleware = headroomMiddleware({ baseUrl: "http://localhost:8787" });
const out = await middleware.transformParams({ params: { prompt }, model: { specificationVersion: "v4" }, type: "generate" });
const body = JSON.parse(mockFetch.mock.calls[0][1].body);
expect(body.messages[0].content[1]).toEqual({
type: "image_url",
image_url: { url: "data:image/png;base64,iVBORw==", part: "file", data_tag: "data" },
});
const restored: LanguageModelV4FilePart = out.prompt[0].content[1];
expect(restored).toEqual({ type: "file", mediaType: "image/png", data: { type: "data", data: "iVBORw==" } });
});
it("passes a provider file reference through transformParams untouched", async () => {
const filePart: LanguageModelV4FilePart = {
type: "file",
mediaType: "application/pdf",
data: { type: "reference", reference: { openai: "file-abc" } },
};
mockFetch.mockResolvedValueOnce(
mockCompressResponse([
{
role: "user",
content: [
{ type: "text", text: "compressed" },
{ type: "image_url", image_url: { url: "data:application/pdf;base64,", part: "file", data_tag: "reference", reference: { openai: "file-abc" } } },
],
},
]),
);
const middleware = headroomMiddleware({ baseUrl: "http://localhost:8787" });
const out = await middleware.transformParams({
params: { prompt: [{ role: "user", content: [{ type: "text", text: "summarise" }, filePart] }] },
model: { specificationVersion: "v4" },
type: "generate",
});
expect(out.prompt[0].content[1]).toEqual(filePart);
});
});
describe("compressVercelMessages", () => {
@@ -205,6 +348,22 @@ describe("compressVercelMessages", () => {
expect(result.messages[1].content[0].toolName).toBe("getWeather");
});
it("does not throw on byte image data", async () => {
mockFetch.mockResolvedValueOnce(
mockCompressResponse([
{ role: "user", content: [{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } }] },
]),
);
await expect(
compressVercelMessages(
[{ role: "user", content: [{ type: "image", image: new Uint8Array([137, 80, 78, 71]), mediaType: "image/png" }] }],
{ baseUrl: "http://localhost:8787" },
),
).resolves.toMatchObject({
messages: [{ role: "user", content: [{ type: "image", image: "iVBORw==", mediaType: "image/png" }] }],
});
});
});
describe("withHeadroom", () => {
+67
View File
@@ -135,3 +135,70 @@ describe("compress()", () => {
expect(body.config.target_ratio).toBe(0.25);
});
});
describe("compress() autodetects Vercel media-only conversations", () => {
beforeEach(() => {
mockFetch.mockReset();
});
it("sends binary file parts as image_url data URIs and restores Vercel file parts", async () => {
mockFetch.mockResolvedValueOnce(
okResponse({
messages: [
{
role: "user",
content: [
{ type: "text", text: "compressed" },
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file" } },
],
},
],
}),
);
const result = await compress(
[
{
role: "user",
content: [
{ type: "text", text: "what is this?" },
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]) },
],
},
],
{ baseUrl: "http://localhost:8787" },
);
const body = JSON.parse(mockFetch.mock.calls[0][1].body);
expect(body.messages[0].content[1]).toEqual({
type: "image_url",
image_url: { url: "data:image/png;base64,iVBORw==", part: "file" },
});
expect(JSON.stringify(body)).not.toContain('"0":137');
expect(result.messages[0].content[1]).toEqual({ type: "file", mediaType: "image/png", data: "iVBORw==" });
});
});
describe("compress() keeps Anthropic text documents through the public path", () => {
beforeEach(() => {
mockFetch.mockReset();
});
it("sends a document-only turn as a media part and restores the document block", async () => {
const doc = {
type: "document",
source: { type: "text", media_type: "text/plain", data: "Critical contract: do not delete account 123" },
title: "Contract",
citations: { enabled: true },
};
// The proxy passes non-text parts through unchanged; echo the request's messages back.
mockFetch.mockImplementationOnce(async (_url: string, init: any) => okResponse({ messages: JSON.parse(init.body).messages }));
const result = await compress([{ role: "user", content: [doc] }], { baseUrl: "http://localhost:8787" });
const body = JSON.parse(mockFetch.mock.calls[0][1].body);
expect(body.messages).toHaveLength(1);
expect(body.messages[0].content[0].type).toBe("image_url");
expect(result.messages).toEqual([{ role: "user", content: [doc] }]);
});
});
+2 -2
View File
@@ -66,9 +66,9 @@ describe("Message types", () => {
>();
});
it("AssistantMessage content can be string or null", () => {
it("AssistantMessage content can be a string, content parts, or null", () => {
expectTypeOf<AssistantMessage["role"]>().toEqualTypeOf<"assistant">();
expectTypeOf<AssistantMessage["content"]>().toEqualTypeOf<string | null>();
expectTypeOf<AssistantMessage["content"]>().toEqualTypeOf<string | ContentPart[] | null>();
});
it("AssistantMessage tool_calls is optional", () => {
@@ -131,3 +131,57 @@ describe("detectFormat", () => {
expect(detectFormat(messages)).toBe("vercel");
});
});
describe("detectFormat: Vercel media parts", () => {
it("detects a media-only Vercel turn with a file part", () => {
expect(
detectFormat([
{
role: "user",
content: [
{ type: "text", text: "what is this?" },
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]) },
],
},
]),
).toBe("vercel");
});
it("detects a media-only Vercel turn with an image part", () => {
expect(
detectFormat([
{ role: "user", content: [{ type: "image", image: new Uint8Array([137, 80, 78, 71]), mediaType: "image/png" }] },
]),
).toBe("vercel");
});
it("still treats OpenAI image_url parts as openai", () => {
expect(
detectFormat([{ role: "user", content: [{ type: "image_url", image_url: { url: "https://example.com/a.png" } }] }]),
).toBe("openai");
});
it("still treats Anthropic image blocks (with a source) as anthropic", () => {
expect(
detectFormat([
{ role: "user", content: [{ type: "image", source: { type: "base64", media_type: "image/png", data: "AAAA" } }] },
]),
).toBe("anthropic");
});
});
describe("detectFormat: Anthropic document blocks", () => {
it("detects a turn whose only Anthropic marker is a document block", () => {
expect(
detectFormat([
{
role: "assistant",
content: [
{ type: "text", text: "Report" },
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: "JVBERi0=" } },
],
},
]),
).toBe("anthropic");
});
});
+621
View File
@@ -6,6 +6,9 @@ import {
openAIToAnthropic,
geminiToOpenAI,
openAIToGemini,
fromOpenAI,
detectFormat,
toOpenAI,
} from "../../src/utils/format.js";
import type { OpenAIMessage } from "../../src/types.js";
@@ -838,3 +841,621 @@ describe("openAIToVercel tool names", () => {
});
});
describe("vercelToOpenAI media parts", () => {
it("converts a provider-prompt file part (image bytes) to an image_url data URI", () => {
const result = vercelToOpenAI([
{
role: "user",
content: [
{ type: "text", text: "what is this?" },
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]) },
],
},
]);
expect(result).toEqual([
{
role: "user",
content: [
{ type: "text", text: "what is this?" },
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file" } },
],
},
]);
});
it("keeps a file-only turn (pdf) and carries its filename", () => {
const result = vercelToOpenAI([
{ role: "user", content: [{ type: "file", mediaType: "application/pdf", data: "JVBERi0=", filename: "a.pdf" }] },
]);
expect(result).toEqual([
{
role: "user",
content: [
{ type: "image_url", image_url: { url: "data:application/pdf;base64,JVBERi0=", part: "file", filename: "a.pdf" } },
],
},
]);
});
it("encodes a Uint8Array image as a data URI instead of String(bytes)", () => {
const result = vercelToOpenAI([
{ role: "user", content: [{ type: "image", image: new Uint8Array([137, 80, 78, 71]), mediaType: "image/png" }] },
]);
expect(result).toEqual([
{ role: "user", content: [{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } }] },
]);
});
it("treats a bare base64 string image as data, not a URL", () => {
const result = vercelToOpenAI([
{ role: "user", content: [{ type: "image", image: "iVBORw==", mediaType: "image/png" }] },
]);
expect(result).toEqual([
{ role: "user", content: [{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } }] },
]);
});
it("carries the MIME type of a URL-backed file part", () => {
const result = vercelToOpenAI([
{ role: "user", content: [{ type: "file", mediaType: "application/pdf", data: new URL("https://example.com/a.pdf") }] },
]);
expect(result).toEqual([
{
role: "user",
content: [
{ type: "image_url", image_url: { url: "https://example.com/a.pdf", part: "file", mime_type: "application/pdf" } },
],
},
]);
});
});
describe("openAIToVercel media parts", () => {
it("restores a data URI marked part=file as a file part with base64 data", () => {
const msgs = [
{
role: "user",
content: [
{ type: "image_url", image_url: { url: "data:application/pdf;base64,JVBERi0=", part: "file", filename: "a.pdf" } },
],
},
] as OpenAIMessage[];
expect(openAIToVercel(msgs)).toEqual([
{ role: "user", content: [{ type: "file", mediaType: "application/pdf", data: "JVBERi0=", filename: "a.pdf" }] },
]);
});
it("restores an image data URI as an image part with base64 data and mediaType", () => {
const msgs: OpenAIMessage[] = [
{ role: "user", content: [{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } }] },
];
expect(openAIToVercel(msgs)).toEqual([
{ role: "user", content: [{ type: "image", image: "iVBORw==", mediaType: "image/png" }] },
]);
});
it("restores a URL-backed file part from the carried MIME type", () => {
const msgs = [
{
role: "user",
content: [
{ type: "image_url", image_url: { url: "https://example.com/a.pdf", part: "file", mime_type: "application/pdf" } },
],
},
] as OpenAIMessage[];
const part = openAIToVercel(msgs)[0].content[0];
expect(part.type).toBe("file");
expect(part.mediaType).toBe("application/pdf");
expect(part.data).toBeInstanceOf(URL);
expect(part.data.toString()).toBe("https://example.com/a.pdf");
});
it("never throws on an unparseable url and keeps it as a string", () => {
const msgs: OpenAIMessage[] = [
{ role: "user", content: [{ type: "image_url", image_url: { url: "137,80,78,71" } }] },
];
expect(() => openAIToVercel(msgs)).not.toThrow();
expect(openAIToVercel(msgs)[0].content[0]).toEqual({ type: "image", image: "137,80,78,71" });
});
});
describe("round-trip: vercelToOpenAI then openAIToVercel (media)", () => {
it("round-trips a provider-prompt text+file turn into a valid file part", () => {
const original = [
{
role: "user",
content: [
{ type: "text", text: "what is this?" },
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]) },
],
},
];
expect(openAIToVercel(vercelToOpenAI(original))).toEqual([
{
role: "user",
content: [
{ type: "text", text: "what is this?" },
{ type: "file", mediaType: "image/png", data: "iVBORw==" },
],
},
]);
});
it("round-trips a byte image without throwing", () => {
const original = [
{ role: "user", content: [{ type: "image", image: new Uint8Array([137, 80, 78, 71]), mediaType: "image/png" }] },
];
expect(openAIToVercel(vercelToOpenAI(original))).toEqual([
{ role: "user", content: [{ type: "image", image: "iVBORw==", mediaType: "image/png" }] },
]);
});
});
describe("vercelToOpenAI / openAIToVercel assistant file parts", () => {
it("keeps an assistant file part (generated image) alongside its text", () => {
const result = vercelToOpenAI([
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]), filename: "cat.png" },
],
},
]);
expect(result).toEqual([
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file", filename: "cat.png" } },
],
},
]);
});
it("restores an assistant content array as text and file parts with the assistant role", () => {
const msgs = [
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==", part: "file", filename: "cat.png" } },
],
},
] as OpenAIMessage[];
expect(openAIToVercel(msgs)).toEqual([
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "file", mediaType: "image/png", data: "iVBORw==", filename: "cat.png" },
],
},
]);
});
it("round-trips an assistant text+file turn", () => {
const original = [
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "file", mediaType: "image/png", data: new Uint8Array([137, 80, 78, 71]) },
],
},
];
expect(openAIToVercel(vercelToOpenAI(original))).toEqual([
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "file", mediaType: "image/png", data: "iVBORw==" },
],
},
]);
});
it("leaves text-only assistant turns as a string (backward compat)", () => {
expect(
vercelToOpenAI([{ role: "assistant", content: [{ type: "text", text: "a" }, { type: "text", text: "b" }] }]),
).toEqual([{ role: "assistant", content: "ab" }]);
});
});
describe("assistant media across formats", () => {
it("openAIToAnthropic keeps an assistant image as an image block after the text", () => {
const msgs = [
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } },
],
},
] as OpenAIMessage[];
expect(openAIToAnthropic(msgs)).toEqual([
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw==" } },
],
},
]);
});
it("openAIToAnthropic maps an assistant PDF to a document block (base64 and url sources)", () => {
const msgs = [
{
role: "assistant",
content: [
{ type: "image_url", image_url: { url: "data:application/pdf;base64,JVBERi0=", part: "file" } },
{ type: "image_url", image_url: { url: "https://example.com/a.pdf", part: "file", mime_type: "application/pdf" } },
],
},
] as OpenAIMessage[];
expect(openAIToAnthropic(msgs)).toEqual([
{
role: "assistant",
content: [
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: "JVBERi0=" } },
{ type: "document", source: { type: "url", url: "https://example.com/a.pdf" } },
],
},
]);
});
it("openAIToAnthropic leaves a text-only assistant array as a string (backward compat)", () => {
const msgs = [{ role: "assistant", content: [{ type: "text", text: "a" }] }] as OpenAIMessage[];
expect(openAIToAnthropic(msgs)).toEqual([{ role: "assistant", content: "a" }]);
});
it("openAIToGemini keeps an assistant image as an inlineData part after the text", () => {
const msgs = [
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } },
],
},
] as OpenAIMessage[];
expect(openAIToGemini(msgs)).toEqual([
{ role: "model", parts: [{ text: "generated image" }, { inlineData: { mimeType: "image/png", data: "iVBORw==" } }] },
]);
});
it("openAIToGemini maps a URL-backed assistant file to fileData with its MIME type", () => {
const msgs = [
{
role: "assistant",
content: [{ type: "image_url", image_url: { url: "https://example.com/a.pdf", part: "file", mime_type: "application/pdf" } }],
},
] as OpenAIMessage[];
expect(openAIToGemini(msgs)).toEqual([
{ role: "model", parts: [{ fileData: { mimeType: "application/pdf", fileUri: "https://example.com/a.pdf" } }] },
]);
});
it("anthropicToOpenAI keeps assistant image and document blocks as ordered parts", () => {
const result = anthropicToOpenAI([
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw==" } },
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: "JVBERi0=" } },
],
},
]);
expect(result).toEqual([
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } },
{ type: "image_url", image_url: { url: "data:application/pdf;base64,JVBERi0=" } },
],
},
]);
});
it("geminiToOpenAI keeps model-turn inlineData as an ordered image_url part", () => {
const result = geminiToOpenAI([
{ role: "model", parts: [{ text: "generated image" }, { inlineData: { mimeType: "image/png", data: "iVBORw==" } }] },
]);
expect(result).toEqual([
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } },
],
},
]);
});
it("round-trips an Anthropic assistant text+image turn exactly", () => {
const original = [
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw==" } },
],
},
];
expect(openAIToAnthropic(anthropicToOpenAI(original))).toEqual(original);
});
it("round-trips a Gemini model text+inlineData turn exactly", () => {
const original = [
{ role: "model", parts: [{ text: "generated image" }, { inlineData: { mimeType: "image/png", data: "iVBORw==" } }] },
];
expect(openAIToGemini(geminiToOpenAI(original))).toEqual(original);
});
it("carries a Vercel assistant file into Gemini and Anthropic media parts", () => {
const openai = vercelToOpenAI([
{ role: "assistant", content: [{ type: "text", text: "generated image" }, { type: "file", mediaType: "image/png", data: "iVBORw==" }] },
]);
expect(fromOpenAI(openai, "gemini")).toEqual([
{ role: "model", parts: [{ text: "generated image" }, { inlineData: { mimeType: "image/png", data: "iVBORw==" } }] },
]);
expect(fromOpenAI(openai, "anthropic")).toEqual([
{
role: "assistant",
content: [
{ type: "text", text: "generated image" },
{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw==" } },
],
},
]);
});
});
describe("Vercel V4 tagged file data", () => {
it("carries a data-tagged file part and restores the tagged shape", () => {
const original = [
{
role: "user",
content: [
{ type: "text", text: "what is this?" },
{ type: "file", mediaType: "image/png", data: { type: "data", data: new Uint8Array([137, 80, 78, 71]) } },
],
},
];
const openai = vercelToOpenAI(original);
expect(openai[0].content[1]).toEqual({
type: "image_url",
image_url: { url: "data:image/png;base64,iVBORw==", part: "file", data_tag: "data" },
});
expect(openAIToVercel(openai)).toEqual([
{
role: "user",
content: [
{ type: "text", text: "what is this?" },
{ type: "file", mediaType: "image/png", data: { type: "data", data: "iVBORw==" } },
],
},
]);
});
it("carries a url-tagged file part and restores a URL object inside the tag", () => {
const original = [
{ role: "user", content: [{ type: "file", mediaType: "application/pdf", data: { type: "url", url: new URL("https://example.com/a.pdf") } }] },
];
const openai = vercelToOpenAI(original);
expect(openai[0].content[0]).toEqual({
type: "image_url",
image_url: { url: "https://example.com/a.pdf", part: "file", mime_type: "application/pdf", data_tag: "url" },
});
const part = openAIToVercel(openai)[0].content[0];
expect(part.type).toBe("file");
expect(part.mediaType).toBe("application/pdf");
expect(part.data.type).toBe("url");
expect(part.data.url).toBeInstanceOf(URL);
expect(part.data.url.toString()).toBe("https://example.com/a.pdf");
});
it("round-trips a text-tagged file part", () => {
const original = [
{ role: "user", content: [{ type: "file", mediaType: "text/plain", data: { type: "text", text: "hello, file" }, filename: "a.txt" }] },
];
expect(openAIToVercel(vercelToOpenAI(original))).toEqual(original);
});
it("preserves a provider reference (tagged and bare) instead of dropping it", () => {
const tagged = [
{ role: "user", content: [{ type: "file", mediaType: "application/pdf", data: { type: "reference", reference: { openai: "file-abc" } } }] },
];
expect(openAIToVercel(vercelToOpenAI(tagged))).toEqual(tagged);
const bare = [{ role: "user", content: [{ type: "file", mediaType: "application/pdf", data: { openai: "file-abc" } }] }];
expect(openAIToVercel(vercelToOpenAI(bare))).toEqual(bare);
});
});
describe("assistant text separators across formats", () => {
it("openAIToAnthropic joins adjacent assistant text parts with a newline into one block", () => {
const msgs = [
{ role: "assistant", content: [{ type: "text", text: "first" }, { type: "text", text: "second" }] },
] as OpenAIMessage[];
expect(openAIToAnthropic(msgs)).toEqual([{ role: "assistant", content: "first\nsecond" }]);
});
it("openAIToAnthropic keeps text parts separate when the array carries media", () => {
const msgs = [
{
role: "assistant",
content: [
{ type: "text", text: "a" },
{ type: "text", text: "b" },
{ type: "image_url", image_url: { url: "data:image/png;base64,iVBORw==" } },
{ type: "text", text: "c" },
],
},
] as OpenAIMessage[];
expect(openAIToAnthropic(msgs)).toEqual([
{
role: "assistant",
content: [
{ type: "text", text: "a" },
{ type: "text", text: "b" },
{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw==" } },
{ type: "text", text: "c" },
],
},
]);
});
it("round-trips Gemini and Anthropic assistant turns with two adjacent text parts before media exactly", () => {
const gemini = [
{ role: "model", parts: [{ text: "Hello " }, { text: "world" }, { inlineData: { mimeType: "image/png", data: "iVBORw==" } }] },
];
expect(openAIToGemini(geminiToOpenAI(gemini))).toEqual(gemini);
const anthropic = [
{
role: "assistant",
content: [
{ type: "text", text: "Hello " },
{ type: "text", text: "world" },
{ type: "image", source: { type: "base64", media_type: "image/png", data: "iVBORw==" } },
],
},
];
expect(openAIToAnthropic(anthropicToOpenAI(anthropic))).toEqual(anthropic);
});
it("openAIToGemini joins adjacent assistant text parts with a newline into one part", () => {
const msgs = [
{ role: "assistant", content: [{ type: "text", text: "first" }, { type: "text", text: "second" }] },
] as OpenAIMessage[];
expect(openAIToGemini(msgs)).toEqual([{ role: "model", parts: [{ text: "first\nsecond" }] }]);
});
});
describe("Anthropic user document blocks", () => {
it("keeps a user document (PDF) beside its text as an image_url data URI", () => {
const result = anthropicToOpenAI([
{
role: "user",
content: [
{ type: "text", text: "Summarize" },
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: "JVBERi0=" } },
],
},
]);
expect(result).toEqual([
{
role: "user",
content: [
{ type: "text", text: "Summarize" },
{ type: "image_url", image_url: { url: "data:application/pdf;base64,JVBERi0=" } },
],
},
]);
});
it("keeps a document-only user turn instead of dropping it", () => {
const result = anthropicToOpenAI([
{ role: "user", content: [{ type: "document", source: { type: "url", url: "https://example.com/a.pdf" } }] },
]);
expect(result).toEqual([
{ role: "user", content: [{ type: "image_url", image_url: { url: "https://example.com/a.pdf", mime_type: "application/pdf" } }] },
]);
});
it("round-trips a user text+document turn exactly through detection and conversion", () => {
const original = [
{
role: "user",
content: [
{ type: "text", text: "Summarize" },
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: "JVBERi0=" } },
],
},
];
expect(detectFormat(original)).toBe("anthropic");
expect(fromOpenAI(toOpenAI(original), "anthropic")).toEqual(original);
});
});
describe("Anthropic document sources (text, content, file) and document fields", () => {
it("keeps a document-only user turn with a text source, title and citations through the public path", () => {
const input = [
{
role: "user",
content: [
{
type: "document",
source: { type: "text", media_type: "text/plain", data: "Critical contract: do not delete account 123" },
title: "Contract",
citations: { enabled: true },
},
],
},
];
expect(detectFormat(input)).toBe("anthropic");
const wire = toOpenAI(input);
expect(wire).toHaveLength(1);
expect((wire[0].content as any)[0].type).toBe("image_url");
expect(fromOpenAI(wire, "anthropic")).toEqual(input);
});
it("round-trips a mixed text + text-document turn with context exactly", () => {
const input = [
{
role: "user",
content: [
{ type: "text", text: "Summarize" },
{ type: "document", source: { type: "text", media_type: "text/plain", data: "clause 1\nclause 2" }, context: "Contract excerpt" },
],
},
];
expect(fromOpenAI(toOpenAI(input), "anthropic")).toEqual(input);
});
it("carries content-source and file-source documents verbatim", () => {
const input = [
{
role: "user",
content: [
{ type: "document", source: { type: "content", content: [{ type: "text", text: "clause 1" }] }, title: "Doc" },
{ type: "document", source: { type: "file", file_id: "file_abc" } },
],
},
];
expect(fromOpenAI(toOpenAI(input), "anthropic")).toEqual(input);
});
it("restores a PDF document's title, context and citations", () => {
const input = [
{
role: "user",
content: [
{
type: "document",
source: { type: "base64", media_type: "application/pdf", data: "JVBERi0=" },
title: "Report",
context: "Q3",
citations: { enabled: true },
},
],
},
];
expect(fromOpenAI(toOpenAI(input), "anthropic")).toEqual(input);
});
it("round-trips an assistant document with a title", () => {
const input = [
{
role: "assistant",
content: [
{ type: "text", text: "see attached" },
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: "JVBERi0=" }, title: "Report" },
],
},
];
expect(fromOpenAI(toOpenAI(input), "anthropic")).toEqual(input);
});
});