feat(cli): import provider profiles from HTTP URLs (#3706)

* feat(cli): import provider profiles from HTTP URLs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(cli): initialize crypto provider for remote profiles Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-10-02 07:34:45 +08:00 · 2026-09-25 06:12:18 +00:00
parent 7a50c0899f
commit c93fd94a5f
6 changed files with 332 additions and 39 deletions
@@ -272,7 +272,7 @@ IDs fail instead of creating source precedence. The gateway treats configured
 interceptors as trusted sources and does not verify signature annotations in
 their profile payloads.

-The CLI exposes reusable profile definitions through `openshell profile`, with `list` and `describe` reading the same effective catalog used by provider creation. Export, import, update, lint, and delete share that top-level command group. Workspace selection and explicit platform scope apply at the existing profile API boundary; `openshell provider` manages credential-bearing instances.
+The CLI exposes reusable profile definitions through `openshell profile`, with `list` and `describe` reading the same effective catalog used by provider creation. Export, import, update, lint, and delete share that top-level command group. Import and lint accept local files, local directories, or a single HTTP or HTTPS URL; the CLI fetches and parses remote content before submitting it to the gateway. Workspace selection and explicit platform scope apply at the existing profile API boundary; `openshell provider` manages credential-bearing instances.

 Each logical gateway request captures the selected sources into one validated,
 immutable effective catalog before deriving provider behavior. Policy layers,
@@ -1702,10 +1702,11 @@ pub async fn provider_profile_import(
    server: &str,
    file: Option<&Path>,
    from: Option<&Path>,
+    url: Option<&str>,
    workspace: &str,
    tls: &TlsOptions,
 ) -> Result<()> {
-    let (items, mut diagnostics) = load_profile_import_items(file, from)?;
+    let (items, mut diagnostics) = load_profile_import_items_with_url(file, from, url).await?;
    if items.is_empty() && diagnostics.is_empty() {
        return Err(miette!("no provider profile files found"));
    }
@@ -1792,10 +1793,11 @@ pub async fn provider_profile_lint(
    server: &str,
    file: Option<&Path>,
    from: Option<&Path>,
+    url: Option<&str>,
    workspace: &str,
    tls: &TlsOptions,
 ) -> Result<()> {
-    let (items, mut diagnostics) = load_profile_import_items(file, from)?;
+    let (items, mut diagnostics) = load_profile_import_items_with_url(file, from, url).await?;
    if items.is_empty() && diagnostics.is_empty() {
        return Err(miette!("no provider profile files found"));
    }
@@ -2137,6 +2139,92 @@ fn load_profile_import_items(
    Ok((items, diagnostics))
 }

+const MAX_REMOTE_PROFILE_BYTES: usize = 1024 * 1024;
+
+async fn load_profile_import_items_with_url(
+    file: Option<&Path>,
+    from: Option<&Path>,
+    url: Option<&str>,
+) -> Result<(
+    Vec<ProviderProfileImportItem>,
+    Vec<ProviderProfileDiagnostic>,
+)> {
+    let Some(source) = url else {
+        return load_profile_import_items(file, from);
+    };
+    let parsed = reqwest::Url::parse(source)
+        .into_diagnostic()
+        .wrap_err("invalid provider profile URL")?;
+    if !matches!(parsed.scheme(), "http" | "https") {
+        return Err(miette!("provider profile URL must use http or https"));
+    }
+    if !parsed.username().is_empty() || parsed.password().is_some() {
+        return Err(miette!("provider profile URL must not contain userinfo"));
+    }
+    let mut display_url = parsed.clone();
+    display_url.set_query(None);
+    display_url.set_fragment(None);
+    let path = parsed.path().to_owned();
+    let format = Path::new(&path).extension().and_then(|ext| ext.to_str());
+    if !matches!(format, Some("yaml" | "yml" | "json")) {
+        return Err(miette!(
+            "provider profile URL must end in .yaml, .yml, or .json"
+        ));
+    }
+    let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
+    let client = reqwest::Client::builder()
+        .timeout(std::time::Duration::from_secs(15))
+        .build()
+        .into_diagnostic()?;
+    let mut response = client
+        .get(parsed)
+        .send()
+        .await
+        .map_err(|err| {
+            miette!(
+                "failed to fetch provider profile from {display_url}: {}",
+                err.without_url()
+            )
+        })?
+        .error_for_status()
+        .map_err(|err| {
+            miette!(
+                "failed to fetch provider profile from {display_url}: {}",
+                err.without_url()
+            )
+        })?;
+    if response
+        .content_length()
+        .is_some_and(|len| len > MAX_REMOTE_PROFILE_BYTES as u64)
+    {
+        return Err(miette!(
+            "provider profile at {display_url} exceeds the 1 MiB download limit"
+        ));
+    }
+    let mut bytes = Vec::new();
+    while let Some(chunk) = response.chunk().await.map_err(|err| {
+        miette!(
+            "failed to read provider profile from {display_url}: {}",
+            err.without_url()
+        )
+    })? {
+        if bytes.len().saturating_add(chunk.len()) > MAX_REMOTE_PROFILE_BYTES {
+            return Err(miette!(
+                "provider profile at {display_url} exceeds the 1 MiB download limit"
+            ));
+        }
+        bytes.extend_from_slice(&chunk);
+    }
+    let input = std::str::from_utf8(&bytes)
+        .into_diagnostic()
+        .wrap_err_with(|| format!("provider profile at {display_url} is not UTF-8"))?;
+    let item = parse_profile_import_item(display_url.as_str(), input, format);
+    Ok(match item {
+        Ok(item) => (vec![item], Vec::new()),
+        Err(diagnostic) => (Vec::new(), vec![diagnostic]),
+    })
+}
+
 fn profile_source_paths(file: Option<&Path>, from: Option<&Path>) -> Result<Vec<PathBuf>> {
    if let Some(file) = file {
        return Ok(vec![file.to_path_buf()]);
@@ -2176,19 +2264,31 @@ fn load_profile_import_item(
            format!("failed to read provider profile file: {err}"),
        )
    })?;
-    let profile = match path.extension().and_then(|ext| ext.to_str()) {
-        Some("yaml" | "yml") => parse_profile_yaml(&input),
-        Some("json") => parse_profile_json(&input),
+    parse_profile_import_item(
+        &source,
+        &input,
+        path.extension().and_then(|ext| ext.to_str()),
+    )
+}
+
+fn parse_profile_import_item(
+    source: &str,
+    input: &str,
+    format: Option<&str>,
+) -> Result<ProviderProfileImportItem, ProviderProfileDiagnostic> {
+    let profile = match format {
+        Some("yaml" | "yml") => parse_profile_yaml(input),
+        Some("json") => parse_profile_json(input),
        _ => {
            return Err(profile_file_diagnostic(
-                &source,
+                source,
                "unsupported provider profile file format".to_string(),
            ));
        }
    }
-    .map_err(|err| profile_file_diagnostic(&source, err.to_string()))?;
+    .map_err(|err| profile_file_diagnostic(source, err.to_string()))?;

-    let pre_lower = profile.validate_before_lowering(&source);
+    let pre_lower = profile.validate_before_lowering(source);
    if let Some(diag) = pre_lower.into_iter().find(|d| d.severity == "error") {
        return Err(ProviderProfileDiagnostic {
            source: diag.source,
@@ -2201,7 +2301,7 @@ fn load_profile_import_item(

    Ok(ProviderProfileImportItem {
        profile: Some(profile.to_proto()),
-        source,
+        source: source.to_string(),
    })
 }

@@ -1105,8 +1105,8 @@ enum ProfileCommands {
        global: bool,
    },

-    /// Import provider profiles from a file or directory.
-    #[command(group = clap::ArgGroup::new("source").required(true).args(["file", "from"]), help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")]
+    /// Import provider profiles from a file, directory, or HTTP URL.
+    #[command(group = clap::ArgGroup::new("source").required(true).args(["file", "from", "url"]), help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")]
    Import {
        /// Profile file to import.
        #[arg(short = 'f', long = "file", value_hint = ValueHint::FilePath)]
@@ -1116,6 +1116,10 @@ enum ProfileCommands {
        #[arg(long = "from", value_hint = ValueHint::DirPath)]
        from: Option<PathBuf>,

+        /// HTTP or HTTPS URL of one YAML or JSON profile.
+        #[arg(long)]
+        url: Option<String>,
+
        /// Import as platform-scoped profiles (ignores --workspace).
        #[arg(long)]
        global: bool,
@@ -1137,7 +1141,7 @@ enum ProfileCommands {
    },

    /// Validate provider profile files without registering them.
-    #[command(group = clap::ArgGroup::new("source").required(true).args(["file", "from"]), help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")]
+    #[command(group = clap::ArgGroup::new("source").required(true).args(["file", "from", "url"]), help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")]
    Lint {
        /// Profile file to lint.
        #[arg(short = 'f', long = "file", value_hint = ValueHint::FilePath)]
@@ -1147,6 +1151,10 @@ enum ProfileCommands {
        #[arg(long = "from", value_hint = ValueHint::DirPath)]
        from: Option<PathBuf>,

+        /// HTTP or HTTPS URL of one YAML or JSON profile.
+        #[arg(long)]
+        url: Option<String>,
+
        /// Lint against platform scope (ignores --workspace).
        #[arg(long)]
        global: bool,
@@ -1209,11 +1217,17 @@ impl ProfileCommands {
                )
                .await?;
            }
-            Self::Import { file, from, global } => {
+            Self::Import {
+                file,
+                from,
+                url,
+                global,
+            } => {
                run::provider_profile_import(
                    endpoint,
                    file.as_deref(),
                    from.as_deref(),
+                    url.as_deref(),
                    profile_workspace(global),
                    tls,
                )
@@ -1223,11 +1237,17 @@ impl ProfileCommands {
                run::provider_profile_update(endpoint, &id, &file, profile_workspace(global), tls)
                    .await?;
            }
-            Self::Lint { file, from, global } => {
+            Self::Lint {
+                file,
+                from,
+                url,
+                global,
+            } => {
                run::provider_profile_lint(
                    endpoint,
                    file.as_deref(),
                    from.as_deref(),
+                    url.as_deref(),
                    profile_workspace(global),
                    tls,
                )
@@ -5022,7 +5042,7 @@ mod tests {
    #[test]
    fn profile_import_and_lint_require_exactly_one_source() {
        for verb in ["import", "lint"] {
-            for source in ["-f", "--from"] {
+            for source in ["-f", "--from", "--url"] {
                let cli = Cli::try_parse_from([
                    "openshell",
                    "profile",
@@ -5032,19 +5052,30 @@ mod tests {
                    "--global",
                ])
                .expect("profile source should parse");
-                let (file, from, global) = match cli.command {
+                let (file, from, url, global) = match cli.command {
                    Some(Commands::Profile {
                        command:
                            Some(
-                                ProfileCommands::Import { file, from, global }
-                                | ProfileCommands::Lint { file, from, global },
+                                ProfileCommands::Import {
+                                    file,
+                                    from,
+                                    url,
+                                    global,
+                                }
+                                | ProfileCommands::Lint {
+                                    file,
+                                    from,
+                                    url,
+                                    global,
+                                },
                            ),
-                    }) => (file, from, global),
+                    }) => (file, from, url, global),
                    other => panic!("unexpected profile command: {other:?}"),
                };
                assert!(global);
                assert_eq!(file.is_some(), source == "-f");
                assert_eq!(from.is_some(), source == "--from");
+                assert_eq!(url.is_some(), source == "--url");
            }
            assert!(Cli::try_parse_from(["openshell", "profile", verb]).is_err());
            assert!(
@@ -4367,12 +4367,26 @@ binaries: [/usr/bin/custom]
    )
    .unwrap();

-    run::provider_profile_lint(&ts.endpoint, Some(&profile_path), None, "default", &ts.tls)
-        .await
-        .expect("profile lint");
-    run::provider_profile_import(&ts.endpoint, Some(&profile_path), None, "default", &ts.tls)
-        .await
-        .expect("profile import");
+    run::provider_profile_lint(
+        &ts.endpoint,
+        Some(&profile_path),
+        None,
+        None,
+        "default",
+        &ts.tls,
+    )
+    .await
+    .expect("profile lint");
+    run::provider_profile_import(
+        &ts.endpoint,
+        Some(&profile_path),
+        None,
+        None,
+        "default",
+        &ts.tls,
+    )
+    .await
+    .expect("profile import");
    let exported_yaml =
        run::provider_profile_export_text(&ts.endpoint, "custom-api", "yaml", "default", &ts.tls)
            .await
@@ -4938,9 +4952,16 @@ binaries: [/usr/bin/yaml-client]
    .unwrap();
    std::fs::write(dir.path().join("notes.txt"), "ignored").unwrap();

-    run::provider_profile_import(&ts.endpoint, None, Some(dir.path()), "default", &ts.tls)
-        .await
-        .expect("profile import --from");
+    run::provider_profile_import(
+        &ts.endpoint,
+        None,
+        Some(dir.path()),
+        None,
+        "default",
+        &ts.tls,
+    )
+    .await
+    .expect("profile import --from");

    run::provider_profile_export(&ts.endpoint, "custom-yaml", "yaml", "default", &ts.tls)
        .await
@@ -4950,6 +4971,108 @@ binaries: [/usr/bin/yaml-client]
        .expect("custom-json should be imported");
 }

+#[tokio::test]
+async fn provider_profile_lint_and_import_from_http_url() {
+    use tokio::io::{AsyncReadExt, AsyncWriteExt};
+
+    let ts = run_server().await;
+    let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
+    let url = format!(
+        "http://{}/remote.yaml?revision=1",
+        listener.local_addr().unwrap()
+    );
+    tokio::spawn(async move {
+        let body = "id: remote-api\ndisplay_name: Remote API\ncategory: other\n";
+        for _ in 0..2 {
+            let (mut stream, _) = listener.accept().await.unwrap();
+            let mut request = [0_u8; 1024];
+            assert!(stream.read(&mut request).await.unwrap() > 0);
+            let response = format!(
+                "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
+                body.len()
+            );
+            stream.write_all(response.as_bytes()).await.unwrap();
+        }
+    });
+
+    run::provider_profile_lint(&ts.endpoint, None, None, Some(&url), "default", &ts.tls)
+        .await
+        .expect("remote profile lint");
+    run::provider_profile_import(&ts.endpoint, None, None, Some(&url), "default", &ts.tls)
+        .await
+        .expect("remote profile import");
+    run::provider_profile_export(&ts.endpoint, "remote-api", "yaml", "default", &ts.tls)
+        .await
+        .expect("remote profile should be imported");
+}
+
+#[tokio::test]
+async fn provider_profile_import_rejects_unsupported_remote_url() {
+    let ts = run_server().await;
+    let err = run::provider_profile_import(
+        &ts.endpoint,
+        None,
+        None,
+        Some("file:///tmp/profile.yaml"),
+        "default",
+        &ts.tls,
+    )
+    .await
+    .expect_err("file URL must fail");
+    assert!(err.to_string().contains("http or https"));
+}
+
+#[tokio::test]
+async fn provider_profile_import_rejects_oversized_http_response() {
+    use tokio::io::{AsyncReadExt, AsyncWriteExt};
+
+    let ts = run_server().await;
+    let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
+    let url = format!("http://{}/large.yaml", listener.local_addr().unwrap());
+    tokio::spawn(async move {
+        let (mut stream, _) = listener.accept().await.unwrap();
+        let mut request = [0_u8; 1024];
+        assert!(stream.read(&mut request).await.unwrap() > 0);
+        stream
+            .write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 1048577\r\nConnection: close\r\n\r\n")
+            .await
+            .unwrap();
+    });
+    let err =
+        run::provider_profile_import(&ts.endpoint, None, None, Some(&url), "default", &ts.tls)
+            .await
+            .expect_err("oversized profile must fail");
+    assert!(err.to_string().contains("1 MiB download limit"));
+}
+
+#[tokio::test]
+async fn provider_profile_import_redacts_url_query_from_http_errors() {
+    use tokio::io::{AsyncReadExt, AsyncWriteExt};
+
+    let ts = run_server().await;
+    let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
+    let url = format!(
+        "http://{}/missing.yaml?token=private-value",
+        listener.local_addr().unwrap()
+    );
+    tokio::spawn(async move {
+        let (mut stream, _) = listener.accept().await.unwrap();
+        let mut request = [0_u8; 1024];
+        assert!(stream.read(&mut request).await.unwrap() > 0);
+        stream
+            .write_all(b"HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\n\r\n")
+            .await
+            .unwrap();
+    });
+    let err =
+        run::provider_profile_import(&ts.endpoint, None, None, Some(&url), "default", &ts.tls)
+            .await
+            .expect_err("missing remote profile must fail");
+    let message = err.to_string();
+    assert!(message.contains("404"));
+    assert!(!message.contains("private-value"));
+}
+
 #[tokio::test]
 async fn provider_profile_import_preserves_advanced_network_policy_fields() {
    let ts = run_server().await;
@@ -4982,9 +5105,16 @@ binaries:
    )
    .unwrap();

-    run::provider_profile_import(&ts.endpoint, Some(&profile_path), None, "default", &ts.tls)
-        .await
-        .expect("profile import");
+    run::provider_profile_import(
+        &ts.endpoint,
+        Some(&profile_path),
+        None,
+        None,
+        "default",
+        &ts.tls,
+    )
+    .await
+    .expect("profile import");

    let mut client = openshell_cli::tls::grpc_client(&ts.endpoint, &ts.tls)
        .await
@@ -5027,10 +5157,16 @@ endpoints:
    .unwrap();
    std::fs::write(dir.path().join("broken.yaml"), "id: [\n").unwrap();

-    let err =
-        run::provider_profile_import(&ts.endpoint, None, Some(dir.path()), "default", &ts.tls)
-            .await
-            .expect_err("profile import --from should fail on parse errors");
+    let err = run::provider_profile_import(
+        &ts.endpoint,
+        None,
+        Some(dir.path()),
+        None,
+        "default",
+        &ts.tls,
+    )
+    .await
+    .expect_err("profile import --from should fail on parse errors");
    assert!(
        err.to_string().contains("provider profile import failed"),
        "unexpected error: {err}"
@@ -5059,9 +5195,16 @@ endpoints:
    .unwrap();
    std::fs::write(dir.path().join("broken.yaml"), "id: [\n").unwrap();

-    let err = run::provider_profile_lint(&ts.endpoint, None, Some(dir.path()), "default", &ts.tls)
-        .await
-        .expect_err("profile lint --from should fail on parse errors");
+    let err = run::provider_profile_lint(
+        &ts.endpoint,
+        None,
+        Some(dir.path()),
+        None,
+        "default",
+        &ts.tls,
+    )
+    .await
+    .expect_err("profile lint --from should fail on parse errors");
    assert!(
        err.to_string().contains("provider profile lint failed"),
        "unexpected error: {err}"
@@ -277,6 +277,19 @@ Import one profile file at platform scope:
 openshell profile import -f providers/github.yaml --global
 ```

+Import a published YAML or JSON profile directly from an HTTP or HTTPS URL:
+
+```shell
+openshell profile lint --url https://example.com/profiles/github.yaml
+openshell profile import --url https://example.com/profiles/github.yaml --global
+```
+
+Review profiles from remote sources before importing them. A profile can grant
+network access and bind credentials to the endpoints and binaries it declares.
+The URL path must end in `.yaml`, `.yml`, or `.json`; query parameters are allowed.
+Remote downloads have a 1 MiB size limit and a 15 second timeout. `--url` is
+mutually exclusive with `-f` and `--from`.
+
 Import all non-recursive `*.yaml`, `*.yml`, and `*.json` files from a directory:

 ```shell
@@ -168,8 +168,14 @@ openshell profile describe github
 openshell profile export github --output yaml
 openshell profile lint --file ./my-profile.yaml
 openshell profile import --file ./my-profile.yaml
+openshell profile lint --url https://example.com/profiles/my-profile.yaml
+openshell profile import --url https://example.com/profiles/my-profile.yaml
 ```

+`--url` accepts one HTTP or HTTPS YAML or JSON profile. Review its endpoint and
+binary grants before importing it. The URL path must end in `.yaml`, `.yml`, or
+`.json`; downloads are limited to 1 MiB and 15 seconds.
+
 Use `profile describe` to inspect a definition's credential metadata, endpoints, TLS handling, MCP access settings, rule counts, binaries, source, and scope before creating a provider. Check for `tls: skip` and the uninspected-credential opt-in before relying on displayed L7 rules. List and describe accept table, JSON, and YAML output; use structured output for complete rule definitions, `--workspace` for a workspace catalog, or `--global` for platform scope. Use `profile export` when preparing an editable definition, `profile update <id> --file <file>` to replace an existing custom profile with its current resource version, and `profile delete <id>...` to remove custom profiles. Provider instances remain under `provider`.

 Existing scripts can continue using `provider list-profiles` and `provider profile export/import/update/lint/delete`. These commands share the top-level handlers and preserve their arguments, output options, and workspace/global flags. Prefer `profile` when writing new commands.