mirror of
https://github.com/rohitg00/ai-engineering-from-scratch.git
synced 2026-10-02 01:54:39 +08:00
feat(phase-17/02): inference platform economics - Fireworks, Together, Baseten, Modal
This commit is contained in:
Rohit Ghumare
+80
@@ -0,0 +1,80 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 960 560" font-family="Georgia, 'Times New Roman', serif">
|
||||
<defs>
|
||||
<style>
|
||||
.box { fill: #faf6ef; stroke: #1a1a1a; stroke-width: 1.5; }
|
||||
.silicon { fill: #ffe1e1; stroke: #c0392b; stroke-width: 1.5; }
|
||||
.gpu { fill: #dfe9ff; stroke: #2c5ea9; stroke-width: 1.5; }
|
||||
.api { fill: #e6f4ea; stroke: #2e7d32; stroke-width: 1.5; }
|
||||
.title { font-size: 16px; font-weight: 700; fill: #1a1a1a; }
|
||||
.head { font-size: 12px; font-weight: 700; fill: #1a1a1a; }
|
||||
.step { font-size: 12px; font-family: 'Menlo', monospace; fill: #222; }
|
||||
.small { font-size: 10px; font-family: 'Menlo', monospace; fill: #555; }
|
||||
.caption { font-size: 11px; fill: #555; font-style: italic; }
|
||||
</style>
|
||||
</defs>
|
||||
<text x="480" y="24" text-anchor="middle" class="title">Inference platform map — three segments, six vendors</text>
|
||||
|
||||
<rect x="40" y="50" width="280" height="460" class="box"/>
|
||||
<text x="180" y="72" text-anchor="middle" class="head">custom silicon</text>
|
||||
<rect x="60" y="90" width="240" height="60" class="silicon"/>
|
||||
<text x="180" y="110" text-anchor="middle" class="step">Groq (LPU)</text>
|
||||
<text x="180" y="128" text-anchor="middle" class="small">voice agents, real-time translation</text>
|
||||
<text x="180" y="144" text-anchor="middle" class="small">5-10x faster decode than GPU</text>
|
||||
<rect x="60" y="160" width="240" height="60" class="silicon"/>
|
||||
<text x="180" y="180" text-anchor="middle" class="step">Cerebras (WSE)</text>
|
||||
<text x="180" y="198" text-anchor="middle" class="small">wafer-scale engine</text>
|
||||
<text x="180" y="214" text-anchor="middle" class="small">very large context decode</text>
|
||||
<rect x="60" y="230" width="240" height="60" class="silicon"/>
|
||||
<text x="180" y="250" text-anchor="middle" class="step">SambaNova (RDU)</text>
|
||||
<text x="180" y="268" text-anchor="middle" class="small">reconfigurable dataflow</text>
|
||||
<text x="180" y="284" text-anchor="middle" class="small">enterprise regulated</text>
|
||||
<text x="180" y="330" text-anchor="middle" class="caption">pay premium per-token</text>
|
||||
<text x="180" y="346" text-anchor="middle" class="caption">win on latency SLAs</text>
|
||||
|
||||
<rect x="340" y="50" width="280" height="460" class="box"/>
|
||||
<text x="480" y="72" text-anchor="middle" class="head">GPU platforms</text>
|
||||
<rect x="360" y="90" width="240" height="60" class="gpu"/>
|
||||
<text x="480" y="110" text-anchor="middle" class="step">Fireworks — latency</text>
|
||||
<text x="480" y="128" text-anchor="middle" class="small">FireAttention, $4B valn</text>
|
||||
<text x="480" y="144" text-anchor="middle" class="small">$1/hr GPU raise May 2026</text>
|
||||
<rect x="360" y="160" width="240" height="60" class="gpu"/>
|
||||
<text x="480" y="180" text-anchor="middle" class="step">Together — breadth</text>
|
||||
<text x="480" y="198" text-anchor="middle" class="small">200+ models, 50-70% under Replicate</text>
|
||||
<text x="480" y="214" text-anchor="middle" class="small">"AI Native Cloud"</text>
|
||||
<rect x="360" y="230" width="240" height="60" class="gpu"/>
|
||||
<text x="480" y="250" text-anchor="middle" class="step">Baseten — enterprise</text>
|
||||
<text x="480" y="268" text-anchor="middle" class="small">$5B valn Jan 2026, SOC2+HIPAA</text>
|
||||
<text x="480" y="284" text-anchor="middle" class="small">Truss, per-min billing</text>
|
||||
<rect x="360" y="300" width="240" height="60" class="gpu"/>
|
||||
<text x="480" y="320" text-anchor="middle" class="step">Modal — Python DX</text>
|
||||
<text x="480" y="338" text-anchor="middle" class="small">$1.1B valn, per-sec billing</text>
|
||||
<text x="480" y="354" text-anchor="middle" class="small">infrastructure-as-Python</text>
|
||||
<rect x="360" y="370" width="240" height="60" class="gpu"/>
|
||||
<text x="480" y="390" text-anchor="middle" class="step">Anyscale — Ray-native</text>
|
||||
<text x="480" y="408" text-anchor="middle" class="small">RayTurbo inference engine</text>
|
||||
<text x="480" y="424" text-anchor="middle" class="small">managed Ray clusters</text>
|
||||
|
||||
<rect x="640" y="50" width="280" height="460" class="box"/>
|
||||
<text x="780" y="72" text-anchor="middle" class="head">API-first marketplaces</text>
|
||||
<rect x="660" y="90" width="240" height="60" class="api"/>
|
||||
<text x="780" y="110" text-anchor="middle" class="step">Replicate</text>
|
||||
<text x="780" y="128" text-anchor="middle" class="small">pay-per-prediction</text>
|
||||
<text x="780" y="144" text-anchor="middle" class="small">multimodal breadth default</text>
|
||||
<rect x="660" y="160" width="240" height="60" class="api"/>
|
||||
<text x="780" y="180" text-anchor="middle" class="step">DeepInfra</text>
|
||||
<text x="780" y="198" text-anchor="middle" class="small">aggressive per-token pricing</text>
|
||||
<text x="780" y="214" text-anchor="middle" class="small">hobbyist and prototyping</text>
|
||||
<rect x="660" y="230" width="240" height="60" class="api"/>
|
||||
<text x="780" y="250" text-anchor="middle" class="step">OpenRouter</text>
|
||||
<text x="780" y="268" text-anchor="middle" class="small">unified API over 100+ models</text>
|
||||
<text x="780" y="284" text-anchor="middle" class="small">single credit for many providers</text>
|
||||
<rect x="660" y="300" width="240" height="60" class="api"/>
|
||||
<text x="780" y="320" text-anchor="middle" class="step">Fal</text>
|
||||
<text x="780" y="338" text-anchor="middle" class="small">image / video focus</text>
|
||||
<text x="780" y="354" text-anchor="middle" class="small">serverless per-call</text>
|
||||
<text x="780" y="420" text-anchor="middle" class="caption">time-to-first-call optimized</text>
|
||||
<text x="780" y="436" text-anchor="middle" class="caption">cheapest for bursty and small-scale</text>
|
||||
|
||||
<rect x="40" y="520" width="880" height="30" class="box"/>
|
||||
<text x="480" y="538" text-anchor="middle" class="step">rule: per-minute beats per-token above ~30% sustained utilization of one GPU</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 5.7 KiB |
@@ -0,0 +1,108 @@
|
||||
"""Inference platform economics comparator — stdlib Python.
|
||||
|
||||
Models six providers (Fireworks, Together, Baseten, Modal, Replicate, Anyscale)
|
||||
on the same synthetic workload. Normalizes per-token vs per-minute vs per-prediction
|
||||
pricing so you can compare head-to-head.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class Vendor:
|
||||
name: str
|
||||
model: str
|
||||
per_mtok_output: float | None # $/M output tokens (None if not the model)
|
||||
per_minute: float | None # $/minute for dedicated GPU (None if serverless)
|
||||
per_prediction: float | None # $/prediction (None if per-token)
|
||||
tokens_per_minute: int # effective tokens when GPU is saturated
|
||||
cold_start_sec: float
|
||||
notes: str
|
||||
|
||||
|
||||
VENDORS = [
|
||||
Vendor("Fireworks", "Llama 70B", 0.90, None, None, 900_000, 1.5, "FireAttention, batch tier 50% off"),
|
||||
Vendor("Together", "Llama 70B", 0.88, None, None, 850_000, 2.0, "200+ models, 50-70% below Replicate"),
|
||||
Vendor("Baseten", "Custom Llama 70B", None, 0.55, None, 900_000, 5.0, "Truss, SOC2 HIPAA, per-min billing"),
|
||||
Vendor("Modal", "Custom Llama 70B", None, 0.48, None, 800_000, 2.5, "Python-native, per-sec billing"),
|
||||
Vendor("Replicate", "Llama 70B", None, None, 0.006, 750_000, 4.0, "Pay-per-prediction, multimodal"),
|
||||
Vendor("Anyscale", "Llama 70B RayTurbo", None, 0.60, None, 850_000, 3.0, "Ray-native, distributed Python"),
|
||||
]
|
||||
|
||||
|
||||
def cost_per_day(v: Vendor, tokens_per_day: int, predictions_per_day: int) -> float:
|
||||
"""Effective $/day given the vendor's pricing model."""
|
||||
if v.per_mtok_output is not None:
|
||||
return (tokens_per_day / 1e6) * v.per_mtok_output
|
||||
if v.per_minute is not None:
|
||||
minutes = tokens_per_day / v.tokens_per_minute
|
||||
return minutes * v.per_minute
|
||||
if v.per_prediction is not None:
|
||||
return predictions_per_day * v.per_prediction
|
||||
return 0.0
|
||||
|
||||
|
||||
def effective_rate(v: Vendor, tokens_per_day: int, predictions_per_day: int) -> float:
|
||||
"""Normalize to $/M tokens for cross-vendor comparison."""
|
||||
c = cost_per_day(v, tokens_per_day, predictions_per_day)
|
||||
return (c / (tokens_per_day / 1e6)) if tokens_per_day else 0
|
||||
|
||||
|
||||
def run_scenario(label: str, tokens_per_day: int, predictions_per_day: int) -> None:
|
||||
print(f"\n{label}")
|
||||
print(f"Workload: {tokens_per_day/1e6:.1f}M output tokens/day | {predictions_per_day} predictions/day")
|
||||
header = f"{'Vendor':12} {'Model':22} {'$/day':>8} {'$/M tok':>10} Notes"
|
||||
print(header)
|
||||
print("-" * len(header))
|
||||
for v in VENDORS:
|
||||
cost = cost_per_day(v, tokens_per_day, predictions_per_day)
|
||||
rate = effective_rate(v, tokens_per_day, predictions_per_day)
|
||||
print(f"{v.name:12} {v.model:22} ${cost:7.2f} ${rate:9.2f} {v.notes}")
|
||||
|
||||
|
||||
def utilization_breakeven() -> None:
|
||||
print("\n" + "=" * 80)
|
||||
print("PER-TOKEN vs PER-MINUTE BREAK-EVEN — Fireworks (per-token) vs Baseten (per-min)")
|
||||
print("=" * 80)
|
||||
fw = VENDORS[0]
|
||||
bt = VENDORS[2]
|
||||
print(f"Fireworks: ${fw.per_mtok_output:.2f}/M output | Baseten: ${bt.per_minute:.2f}/min, {bt.tokens_per_minute/1e3:.0f}k tok/min\n")
|
||||
print(f"{'Util %':>8} {'Fireworks $/day':>16} {'Baseten $/day':>14} Winner")
|
||||
for util_pct in (5, 10, 15, 20, 25, 30, 35, 40, 50, 75, 100):
|
||||
tokens_per_day = int(bt.tokens_per_minute * 60 * 24 * util_pct / 100)
|
||||
fw_cost = cost_per_day(fw, tokens_per_day, 0)
|
||||
bt_cost = 24 * 60 * bt.per_minute
|
||||
winner = "Baseten" if bt_cost < fw_cost else "Fireworks"
|
||||
print(f"{util_pct:>7}% ${fw_cost:>15.2f} ${bt_cost:>13.2f} {winner}")
|
||||
|
||||
|
||||
def cold_start_penalty() -> None:
|
||||
print("\n" + "=" * 80)
|
||||
print("COLD START PENALTY — bursty workload")
|
||||
print("=" * 80)
|
||||
print(f"{'Vendor':12} {'Cold start':>11} Impact at 100 cold invocations/day")
|
||||
for v in VENDORS:
|
||||
impact_sec = v.cold_start_sec * 100
|
||||
print(f"{v.name:12} {v.cold_start_sec:>8.1f} s +{impact_sec:.0f} seconds/day of extra latency")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
print("=" * 80)
|
||||
print("INFERENCE PLATFORM ECONOMICS — 2026 approximations")
|
||||
print("=" * 80)
|
||||
|
||||
run_scenario("Scenario A — startup-scale LLM product",
|
||||
tokens_per_day=2_000_000, predictions_per_day=10_000)
|
||||
run_scenario("Scenario B — high-volume production",
|
||||
tokens_per_day=100_000_000, predictions_per_day=500_000)
|
||||
|
||||
utilization_breakeven()
|
||||
cold_start_penalty()
|
||||
|
||||
print("\nRule of thumb: per-minute (Baseten, Modal) beats per-token above ~30% util.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,131 @@
|
||||
# Inference Platform Economics — Fireworks, Together, Baseten, Modal, Replicate, Anyscale
|
||||
|
||||
> The 2026 inference market is no longer GPU time rental. It bifurcates into custom silicon (Groq, Cerebras, SambaNova), GPU platforms (Baseten, Together, Fireworks, Modal), and API-first marketplaces (Replicate, DeepInfra). Fireworks raised price $1/hr per GPU on May 1, 2026, and $4B valuation on 10T+ tokens/day tells you the volume-driven model works. Baseten closed $300M Series E at $5B in January 2026. The competitive positioning rule is simple: Fireworks optimizes latency, Together optimizes catalog breadth, Baseten optimizes enterprise polish, Modal optimizes Python-native DX, Replicate optimizes multimodal reach, Anyscale optimizes distributed Python. This lesson gives you a matrix you can hand a founder.
|
||||
|
||||
**Type:** Learn
|
||||
**Languages:** Python (stdlib, toy per-call economics comparator)
|
||||
**Prerequisites:** Phase 17 · 01 (Managed LLM Platforms), Phase 17 · 04 (vLLM Serving Internals)
|
||||
**Time:** ~60 minutes
|
||||
|
||||
## Learning Objectives
|
||||
|
||||
- Name the three market segments (custom silicon, GPU platforms, API-first) and map each vendor to a segment.
|
||||
- Explain why the "per-token" API pricing model compresses toward the serving engine's cost curve, not the hardware's.
|
||||
- Compute effective cost per request across at least three vendors and explain when per-minute (Baseten, Modal) beats per-token.
|
||||
- Identify which platform is the right default for a given workload (serverless bursty, steady high-throughput, fine-tuned variants, multimodal).
|
||||
|
||||
## The Problem
|
||||
|
||||
You evaluated managed hyperscaler platforms. You decided you need a narrower, faster provider — Fireworks for latency, Together for breadth, Baseten for a fine-tuned custom model. Now you have six real choices and the pricing pages do not line up. Fireworks shows $/M tokens; Baseten shows $/minute; Modal shows $/second; Replicate shows $/prediction. You cannot compare them head-to-head without modeling the workload.
|
||||
|
||||
Worse, the business model behind each pricing page is different. Fireworks runs its own custom engine (FireAttention) on shared GPUs; the per-token rate reflects their utilization curve. Baseten gives you Truss + dedicated GPUs; per-minute reflects exclusivity. Modal is true Python serverless — per-second billing with sub-second cold starts. Same output (an LLM response), three different cost functions.
|
||||
|
||||
This lesson models the six and tells you when each wins.
|
||||
|
||||
## The Concept
|
||||
|
||||
### The three segments
|
||||
|
||||
**Custom silicon** — Groq (LPU), Cerebras (WSE), SambaNova (RDU). Typically 5-10x faster decode than a GPU-based cluster on the same model. Higher per-token price (Groq was ~$0.99/M on Llama-70B late 2025) but unbeatable for latency-sensitive use cases. Groq is the production pick for voice agents and real-time translation.
|
||||
|
||||
**GPU platforms** — Baseten, Together, Fireworks, Modal, Anyscale. Run on NVIDIA (H100, H200, B200 in 2026) or sometimes AMD. The economic layer between "raw GPU rental" (RunPod, Lambda) and "hyperscaler managed service" (Bedrock).
|
||||
|
||||
**API-first marketplaces** — Replicate, DeepInfra, OpenRouter, Fal. Broad catalog, pay-per-prediction or pay-per-second, emphasize time-to-first-call.
|
||||
|
||||
### Fireworks — latency-optimized GPU platform
|
||||
|
||||
- FireAttention engine (custom); marketed as 4x lower latency than vLLM on equivalent configs.
|
||||
- Batch tier at ~50% of serverless rate for non-interactive workloads.
|
||||
- Fine-tuned model served at the same rate as the base model — a real differentiator versus providers that charge a premium for your LoRA.
|
||||
- Mid-2026: raised on-demand GPU rental $1/hour effective May 1, 2026. Volume pricing negotiable at scale.
|
||||
- Financial signal: $4B valuation, 10T+ tokens/day handled.
|
||||
|
||||
### Together — breadth-optimized
|
||||
|
||||
- 200+ models including open-source releases within days of upstream publication.
|
||||
- 50-70% cheaper than Replicate on equivalent LLM models — the "AI Native Cloud" positioning is volume and catalog.
|
||||
- Inference + fine-tuning + training in one API.
|
||||
|
||||
### Baseten — enterprise-polish-optimized
|
||||
|
||||
- Truss framework: model packaging with dependencies, secrets, serving config in one manifest.
|
||||
- GPU range from T4 through B200. Per-minute billing with reasonable cold-start mitigation.
|
||||
- SOC 2 Type II, HIPAA-ready. Common fintech and healthcare pick.
|
||||
- $5B valuation, January 2026 Series E ($300M from CapitalG, IVP, NVIDIA).
|
||||
|
||||
### Modal — Python-native-optimized
|
||||
|
||||
- Infrastructure-as-code in pure Python. Decorate a function with `@modal.function(gpu="A100")` and deploy with one command.
|
||||
- Per-second billing. Cold starts 2-4s with pre-warming; <1s for small models.
|
||||
- $87M Series B at $1.1B valuation (2025). Strongest developer experience score in independent surveys.
|
||||
|
||||
### Replicate — multimodal breadth
|
||||
|
||||
- Pay-per-prediction. The default platform for image, video, and audio models.
|
||||
- Integration ecosystem (Zapier, Vercel, CMS plugins).
|
||||
- Less competitive on LLM per-token rates but wins on multimodal variety.
|
||||
|
||||
### Anyscale — Ray-native
|
||||
|
||||
- Built on Ray; RayTurbo is Anyscale's proprietary inference engine (competes with vLLM).
|
||||
- Best for distributed Python workloads where the inference step is one node in a larger graph.
|
||||
- Managed Ray clusters; tight integration with Ray AIR and Ray Serve.
|
||||
|
||||
### Per-token versus per-minute — when each wins
|
||||
|
||||
Per-token makes sense when the workload is latency-insensitive and bursty — you only pay for what you use. Per-minute makes sense when utilization is high and predictable — you beat per-token once you're saturating the GPU.
|
||||
|
||||
Rough rule: for workloads above ~30% sustained utilization of a dedicated GPU, per-minute (Baseten, Modal) starts to beat per-token (Fireworks, Together). Below that, per-token wins because you avoid paying for idle.
|
||||
|
||||
### Custom engine is the real moat
|
||||
|
||||
Every platform above vLLM and SGLang claims a custom engine. FireAttention, RayTurbo, Baseten's inference stack. Custom-engine claims shade marketing — the honest framing is that vLLM + SGLang represent about 80% of production open-source inference, and the differentiators at the platform layer are DX, attribution, and SLAs.
|
||||
|
||||
### Numbers you should remember
|
||||
|
||||
- Fireworks GPU rental: $1/hr raise effective May 1, 2026.
|
||||
- Fireworks claim: 4x lower latency than vLLM on equivalent configs.
|
||||
- Together: 50-70% cheaper than Replicate on LLMs.
|
||||
- Baseten valuation: $5B (Series E, Jan 2026, $300M round).
|
||||
- Modal valuation: $1.1B (Series B, 2025).
|
||||
- Per-minute beats per-token above ~30% sustained utilization.
|
||||
|
||||
## Use It
|
||||
|
||||
`code/main.py` compares the six vendors on a synthetic workload across pricing models. Reports $/day and effective $/M tokens. Run it to find the break-even between per-token and per-minute.
|
||||
|
||||
## Ship It
|
||||
|
||||
This lesson produces `outputs/skill-inference-platform-picker.md`. Given workload profile, SLA, and budget, picks the primary inference platform and names the runner-up.
|
||||
|
||||
## Exercises
|
||||
|
||||
1. Run `code/main.py`. At what sustained utilization does Baseten (per-minute) beat Fireworks (per-token) for a 70B model on one H100? Derive the crossover yourself and compare to the rule of thumb.
|
||||
2. Your product serves image generation plus chat plus speech-to-text. Pick platforms for each modality and name the gateway pattern that unifies them.
|
||||
3. Fireworks raises prices by $1/hr on your primary model. Model the blended cost impact if 40% of your traffic moves to batch tier (50% off).
|
||||
4. A regulated customer requires SOC 2 Type II + HIPAA + dedicated GPUs. Which three platforms are viable and which one wins on FinOps?
|
||||
5. Compare cost per 1,000 predictions for Llama 3.1 70B on Fireworks serverless, Together on-demand, Baseten dedicated, and Replicate API. Which is cheapest at 10 predictions/day? At 10,000?
|
||||
|
||||
## Key Terms
|
||||
|
||||
| Term | What people say | What it actually means |
|
||||
|------|----------------|------------------------|
|
||||
| Custom silicon | "non-GPU chips" | Groq LPU, Cerebras WSE, SambaNova RDU — optimized for decode |
|
||||
| FireAttention | "Fireworks engine" | Custom attention kernel; marketed at 4x lower latency than vLLM |
|
||||
| Truss | "Baseten's format" | Model packaging manifest; dependencies + secrets + serving config |
|
||||
| Per-token | "API pricing" | Charge by tokens consumed; pay for no idle |
|
||||
| Per-minute | "dedicated pricing" | Charge by wall-clock GPU time; wins at high utilization |
|
||||
| Per-prediction | "Replicate pricing" | Charge per model invocation; common for image/video |
|
||||
| RayTurbo | "Anyscale engine" | Proprietary inference on Ray; competes with vLLM on Ray clusters |
|
||||
| Batch tier | "50% off" | Non-interactive queue at reduced rate; common on Fireworks, OpenAI |
|
||||
| Fine-tuned at base rate | "Fireworks LoRA" | Charge LoRA-served requests at base model's rate (differentiator) |
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Fireworks Pricing](https://fireworks.ai/pricing) — per-token rates, batch tier, GPU rental.
|
||||
- [Baseten Pricing](https://www.baseten.co/pricing/) — per-minute rates, committed capacity, enterprise tiers.
|
||||
- [Modal Pricing](https://modal.com/pricing) — per-second GPU rates and free tier.
|
||||
- [Together AI Pricing](https://www.together.ai/pricing) — model catalog and per-token rates.
|
||||
- [Anyscale Pricing](https://www.anyscale.com/pricing) — RayTurbo and managed Ray pricing.
|
||||
- [Northflank — Fireworks AI Alternatives](https://northflank.com/blog/7-best-fireworks-ai-alternatives-for-inference) — comparative assessment.
|
||||
- [Infrabase — AI Inference API Providers 2026](https://infrabase.ai/blog/ai-inference-api-providers-compared) — vendor landscape.
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
---
|
||||
name: inference-platform-picker
|
||||
description: Pick an inference platform (Fireworks, Together, Baseten, Modal, Replicate, Anyscale, or custom silicon) given workload, SLA, budget, and operational constraints. Normalize per-token, per-minute, and per-prediction pricing.
|
||||
version: 1.0.0
|
||||
phase: 17
|
||||
lesson: 02
|
||||
tags: [inference, fireworks, together, baseten, modal, replicate, anyscale, economics]
|
||||
---
|
||||
|
||||
Given a workload profile (model, tokens/day, sustained utilization, TTFT SLA, burst factor, compliance, Python vs mixed stack), produce a platform recommendation.
|
||||
|
||||
Produce:
|
||||
|
||||
1. Primary platform. Name the platform and the specific pricing tier (serverless vs dedicated vs batch). Justify with the workload characteristics that match — e.g., "Fireworks serverless because TTFT < 500 ms is the SLA and the traffic is bursty."
|
||||
2. Effective cost. Normalize the chosen pricing model to $/M output tokens. Compare to at least two alternatives. Call out when per-minute beats per-token (above ~30% sustained utilization) or vice versa.
|
||||
3. Cold-start plan. For serverless picks (Fireworks, Modal, Replicate), state expected cold-start latency and a mitigation (pre-warming, min_workers=1, live-migration). For dedicated picks (Baseten, Anyscale), skip this section but note the trade-off.
|
||||
4. Runner-up. Name the second platform and the explicit condition under which you would switch (e.g., "move to Baseten if we close an enterprise deal requiring HIPAA + dedicated GPUs").
|
||||
5. Gateway layer. Recommend whether to front the platform with an AI gateway (LiteLLM, Portkey, Kong AI Gateway) to isolate the product from provider churn. Default: yes, unless scale is below 500 RPS.
|
||||
|
||||
Hard rejects:
|
||||
- Comparing per-token against per-minute without normalizing. Refuse and insist on effective $/M tokens.
|
||||
- Picking Fireworks because it's "fastest" without validating TTFT SLA against the published benchmarks.
|
||||
- Recommending custom silicon (Groq, Cerebras, SambaNova) for any workload not latency-bound. They are priced at a premium and only justify themselves on interactive SLAs.
|
||||
|
||||
Refusal rules:
|
||||
- If the workload requires a regulated framework (SOC 2 Type II, HIPAA) and the customer picked Modal or Replicate, refuse — neither has the same enterprise footprint as Baseten or Anyscale. Suggest Baseten.
|
||||
- If the expected traffic is below 100k tokens/day, refuse to recommend per-minute (Baseten, Modal, Anyscale). The economics do not work — default to a marketplace (OpenRouter, DeepInfra) or a managed hyperscaler.
|
||||
- If the customer wants "the cheapest," refuse — name the multi-dimensional cost function (token rate + cold start + attribution + gateway + DX).
|
||||
|
||||
Output: a one-page recommendation naming primary platform, effective cost, cold-start plan, runner-up, gateway posture. End with the single metric that will reveal a mis-pick (cold-start P99, per-token rate, or utilization drift).
|
||||
Reference in New Issue
Block a user