From bd928b3ebad16c6aedbdec6feaa50882fc105ec1 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Tue, 13 Oct 2026 22:31:07 +0200 Subject: [PATCH] feat(infra): CONNECTS_TO -- a service address in a container's environment A workload's environment names the services it talks to (PRODUCT_CATALOG_SERVICE_ADDR: productcatalogservice:3550, a URL, a ..svc name); the graph had no edge for it, so the service topology of an application was invisible. A Kubernetes workload or a compose service now CONNECTS_TO the Service its containers' environment addresses, with {via: env, key, container} -- never the value: - the address: the host of a URL or of host:port, a ..svc name in the value, or the whole value when the key names an address (*_ADDR, *_ADDRESS, *_HOST, *_URL, *_ENDPOINT, *_URI) and it is a bare DNS name; a templated value, localhost or an IP addresses nothing, and only service DNS forms count (svc, svc.ns, svc.ns.svc...), never another domain's host (cbm_env_address, shared by both sides); - Kubernetes: a reference resolved like every other one -- a Service of the same bundle, in the named namespace or the object's own; - compose: a service of the same file; a variable without a value takes the project's .env, whose values compose expands against the lines before them (CART_ADDR=cart:${CART_PORT}) -- the dotenv reader does that now too. Consumers: trace_path's topology mode, the CLI's edge list and the README (which also gains DEPLOYS and the env join of the previous commit). Measured on 17 infrastructure repositories: 129 edges, every one correct by hand; an independent reading of the values finds 133 pairs repo-wide, the 5 without an edge belong to another bundle or namespace. Signed-off-by: Martin Vogel --- README.md | 5 +- internal/cbm/cbm.h | 7 +- internal/cbm/extract_k8s.c | 27 +++++ internal/cbm/helpers.c | 164 +++++++++++++++++++++++++++++++ internal/cbm/helpers.h | 11 +++ src/cli/cli.c | 4 +- src/mcp/mcp.c | 9 +- src/pipeline/pass_ci.c | 46 +++++++-- src/pipeline/pass_docker.c | 24 ++++- src/pipeline/pass_k8s.c | 33 +++++-- src/pipeline/pipeline_internal.h | 9 +- tests/test_grammar_probe_f.c | 90 +++++++++++++++++ 12 files changed, 402 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index 4f291b488..e64318752 100644 --- a/README.md +++ b/README.md @@ -250,10 +250,11 @@ The install script placed beside the binary is **reported, not deleted** — uni - `HTTP_CALLS`, `ASYNC_CALLS` (cross-service) - `SPAWNS` — the caller starts another program (`subprocess.run`, `exec.Command`, `posix_spawn`, `Process.Start`, a shell command or Make recipe running a program by path, ...); the target is a `Process` node named after the program, or `` when the program is not a literal. When a shell script, Make recipe, GitHub Actions step or Prow job runs a script of the repository (`./hack/x.sh`, `bash hack/x.sh`), the target is that script's `Module` (else its `File`); `make TARGET` (also `$(MAKE)`, `-C`, `-f`) targets the rule of that target, followed through included makefiles; edges from CI carry `ci` and `job`. A Dockerfile's last stage `ENTRYPOINT` (else `CMD`) spawns what the container runs (`api: dockerfile`): the repository script or source copied into the image, the Go `main` of a binary a stage builds with `go build -o`, else a `Process` - `INFRA_MAPS` — a Kubernetes Service to each workload whose pod labels carry every `key: value` of the Service's selector, within one namespace +- `CONNECTS_TO` — a Kubernetes workload or compose service to the Service its containers' environment addresses (`PRODUCT_CATALOG_SERVICE_ADDR: productcatalogservice:3550`, a URL's host, `..svc...`, a bare name on an `*_ADDR` / `*_HOST` / `*_URL` / ... key; compose: the project's `.env` interpolated): a Kubernetes Service of the same bundle and namespace, or a compose service of the same file. The edge keeps `via: env`, `key` and `container` — never the value - `USAGE` with `ref: image` — a Kubernetes workload (any pod template: containers, initContainers, CronJobs, custom resources) or a docker compose service to the `Image` it runs - `DEPLOYS` — what puts manifests, kustomizations or charts on a cluster, to what it puts there: a Flux Kustomization or Argo Application to the kustomizations / manifests / chart of its path, a Flux HelmRelease to its chart (`spec.chart` or `chartRef` → OCIRepository), a shell command or Make recipe (`kubectl apply -f/-k`, `kustomize build | kubectl apply`, `helm install/upgrade/template`), a Go installer loading a chart with the Helm SDK, a Terraform `helm_release`. A chart outside the repository is a shared external `Chart`; the edge says `via` (`flux`, `argo`, `helmrelease`, `kubectl`, `kustomize`, `helm`, `helm-sdk`, `terraform`, ...) - `DEPENDS_ON` — a Helm chart to the charts it declares (the local chart for a resolvable `file://` repository, otherwise an external `Chart`; the edge keeps `repository`), and a `go.mod` / `requirements.txt` file to an external `Package`; a Dockerfile to the `Image`s it builds on (`via: from`, `copy-from`); an `Image` to the Dockerfile it is built from (`via: build`; `tool`: `skaffold`, `compose`, `docker-build` for `docker` / `podman` / `nerdctl` / `buildah` build commands in scripts, Make and CI); a compose service to the services its `depends_on` / `links` name -- `CONFIGURES` — code reading an environment variable to the shared `EnvVar` of that name (`strategy: env_access`); a Kubernetes container, compose service or Dockerfile (`ENV` of the image) setting one to that container's own `EnvVar` (`strategy: env_set`, with `source`: `value`, `configMapKeyRef`, `secretKeyRef`, `fieldRef`, ..., `compose`, `dockerfile` — values are never stored) +- `CONFIGURES` — code reading an environment variable to the shared `EnvVar` of that name (`strategy: env_access`); a Kubernetes container, compose service or Dockerfile (`ENV` of the image) setting one to that container's own `EnvVar` (`strategy: env_set`, with `source`: `value`, `configMapKeyRef`, `secretKeyRef`, `fieldRef`, ..., `compose`, `dockerfile` — values are never stored); and that container's `EnvVar` to the code reading the name inside the files its image is built from (`strategy: env_read`, with `dockerfile`) — the image's files are the sources its Dockerfile's `COPY` / `ADD` put into it, less the context's `.dockerignore` - `EMITS`, `LISTENS_ON` (channels) - `DATA_FLOWS` with arg-to-param mapping + field access chains - `SIMILAR_TO` (MinHash + LSH near-clone detection, Jaccard scored) @@ -765,7 +766,7 @@ mode. `index_status` keeps describing the published graph and its freshness. ### Edge Types -`CONTAINS_FOLDER`, `CONTAINS_FILE`, `DEFINES`, `DEFINES_METHOD`, `IMPORTS`, `CALLS`, `CALL_REFERENCE`, `USAGE`, `INHERITS`, `IMPLEMENTS`, `OVERRIDE`, `DECORATES`, `RAISES`, `THROWS`, `WRITES`, `DATA_FLOWS`, `HTTP_CALLS`, `GRPC_CALLS`, `ASYNC_CALLS`, `EMITS`, `LISTENS_ON`, `HANDLES`, `SPAWNS`, `CONFIGURES`, `INFRA_MAPS`, `DEPENDS_ON`, `REFERENCES_FILE`, `MENTIONS`, `TESTS`, `TESTS_FILE`, `SIMILAR_TO`, `SEMANTICALLY_RELATED`, `FILE_CHANGES_WITH`, `HAS_BRANCH`, and the cross-repo `CROSS_HTTP_CALLS`, `CROSS_GRPC_CALLS`, `CROSS_GRAPHQL_CALLS`, `CROSS_TRPC_CALLS`, `CROSS_ASYNC_CALLS`, `CROSS_CHANNEL` +`CONTAINS_FOLDER`, `CONTAINS_FILE`, `DEFINES`, `DEFINES_METHOD`, `IMPORTS`, `CALLS`, `CALL_REFERENCE`, `USAGE`, `INHERITS`, `IMPLEMENTS`, `OVERRIDE`, `DECORATES`, `RAISES`, `THROWS`, `WRITES`, `DATA_FLOWS`, `HTTP_CALLS`, `GRPC_CALLS`, `ASYNC_CALLS`, `EMITS`, `LISTENS_ON`, `HANDLES`, `SPAWNS`, `CONFIGURES`, `INFRA_MAPS`, `CONNECTS_TO`, `DEPLOYS`, `DEPENDS_ON`, `REFERENCES_FILE`, `MENTIONS`, `TESTS`, `TESTS_FILE`, `SIMILAR_TO`, `SEMANTICALLY_RELATED`, `FILE_CHANGES_WITH`, `HAS_BRANCH`, and the cross-repo `CROSS_HTTP_CALLS`, `CROSS_GRPC_CALLS`, `CROSS_GRAPHQL_CALLS`, `CROSS_TRPC_CALLS`, `CROSS_ASYNC_CALLS`, `CROSS_CHANNEL` `MENTIONS` links a documented definition to the code its doc comment references (C# ``, ``, ``, ``), one edge per pair with `via`, `syntax`, `tier` (`exact` or `unique`), first `line` and `count`. A reference is bound only when it names exactly one definition under the language's own lookup rules; the rest are counted by reason in `index_status` under `doc_links` (samples with `diagnostics='full'`). diff --git a/internal/cbm/cbm.h b/internal/cbm/cbm.h index a855290e1..8bb004037 100644 --- a/internal/cbm/cbm.h +++ b/internal/cbm/cbm.h @@ -701,8 +701,10 @@ typedef struct { /* A fact a Kubernetes manifest states about other objects (extract_k8s.c), * resolved by the K8s pass against the objects of the source's bundle. * CBM_K8S_FACT_REF: object def_index names a kind/ns/name object at a - * reference path (via), drawn as edge (USAGE, INFRA_MAPS or DEPENDS_ON); - * ns NULL = the source's own namespace. + * reference path (via), drawn as edge (USAGE, INFRA_MAPS, DEPENDS_ON or + * CONNECTS_TO); ns NULL = the source's own namespace. A CONNECTS_TO + * reference (via "env") is the Service the value of environment variable + * `key` of container `container` addresses -- the value is not kept. * CBM_K8S_FACT_CRD: the CRD def_index declares kind `kind` in group `name`. * CBM_K8S_FACT_DEPLOY: object def_index (a Flux Kustomization, an Argo * Application) deploys the repository directory `name` into namespace @@ -737,6 +739,7 @@ typedef struct { const char *edge; const char *route; const char *container; + const char *key; /* CONNECTS_TO: the environment variable addressing it */ int line; } CBMK8sFact; diff --git a/internal/cbm/extract_k8s.c b/internal/cbm/extract_k8s.c index 6f051572d..fe913845b 100644 --- a/internal/cbm/extract_k8s.c +++ b/internal/cbm/extract_k8s.c @@ -1064,6 +1064,32 @@ static const char *k8s_env_source(k8s_walk_t *w, TSNode item) { return "value"; } +// The Service a container's literal env value addresses (`host:port`, a URL, +// `..svc...`, a bare name on an address key): a CONNECTS_TO +// reference, resolved in the bundle like every other. Only the service's +// name and namespace are kept -- never the value. +static void k8s_env_address(k8s_walk_t *w, TSNode item, const char *key, const char *cname) { + enum { K8S_DNS_LABEL = 64 }; + const char *value = k8s_text(w, item, "value"); + char svc[K8S_DNS_LABEL]; + char ns[K8S_DNS_LABEL]; + if (!value || cbm_env_address(key, value, svc, sizeof(svc), ns, sizeof(ns)) == 0) { + return; + } + CBMArena *a = w->ctx->arena; + CBMK8sFact f = {.fact = CBM_K8S_FACT_REF, + .def_index = w->def_index, + .kind = "Service", + .ns = ns[0] ? cbm_arena_strdup(a, ns) : NULL, + .name = cbm_arena_strdup(a, svc), + .via = "env", + .edge = "CONNECTS_TO", + .container = cname, + .key = key, + .line = (int)ts_node_start_point(item).row + SKIP_ONE}; + cbm_k8sfact_push(&w->ctx->result->k8s_facts, a, f); +} + // A container (`path` ends in containers[], initContainers[] or // ephemeralContainers[], any depth: pod templates, CronJobs, the pod // templates of custom resources): its literal image, and the environment @@ -1108,6 +1134,7 @@ static void k8s_container(k8s_walk_t *w, const char *path, TSNode container) { .via = k8s_env_source(w, item), .line = (int)ts_node_start_point(item).row + SKIP_ONE}; cbm_k8sfact_push(&w->ctx->result->k8s_facts, w->ctx->arena, f); + k8s_env_address(w, item, key, cname); } } diff --git a/internal/cbm/helpers.c b/internal/cbm/helpers.c index f08d047e7..0a770804f 100644 --- a/internal/cbm/helpers.c +++ b/internal/cbm/helpers.c @@ -2140,6 +2140,170 @@ static bool is_env_var_pattern(const char *s, int len) { return has_upper && has_underscore; } +/* ── Service addresses in environment values ──────────────────────── */ + +static bool env_addr_key(const char *key) { + static const char *const SUFFIXES[] = {"_ADDR", "_ADDRESS", "_HOST", + "_URL", "_ENDPOINT", "_URI"}; + size_t kl = strlen(key); + for (size_t i = 0; i < sizeof(SUFFIXES) / sizeof(SUFFIXES[0]); i++) { + size_t sl = strlen(SUFFIXES[i]); + if (kl >= sl && strcmp(key + kl - sl, SUFFIXES[i]) == 0) { + return true; + } + } + return false; +} + +static bool env_label_char(char c) { + return islower((unsigned char)c) || isdigit((unsigned char)c) || c == '-'; +} + +/* A bare DNS name: `[a-z0-9][a-z0-9-]*(\.[a-z0-9-]+)*`, not only digits. */ +static bool env_bare_name(const char *v) { + bool digits_only = true; + if (!env_label_char(v[0]) || v[0] == '-') { + return false; + } + for (const char *p = v; *p; p++) { + if (*p == '.' ? (p[SKIP_ONE] == '\0' || p[SKIP_ONE] == '.') : !env_label_char(*p)) { + return false; + } + digits_only = digits_only && (isdigit((unsigned char)*p) || *p == '.'); + } + return !digits_only; +} + +static bool env_copy(char *out, size_t sz, const char *s, size_t n) { + if (n == 0 || n >= sz) { + return false; + } + memcpy(out, s, n); + out[n] = '\0'; + return true; +} + +/* The host of a URL value (after any user@, before :port). */ +static bool env_url_host(const char *v, char *host, size_t sz) { + const char *q = v; + if (!islower((unsigned char)*q)) { + return false; + } + while (islower((unsigned char)*q) || isdigit((unsigned char)*q) || *q == '+' || *q == '.' || + *q == '-') { + q++; + } + if (strncmp(q, "://", strlen("://")) != 0) { + return false; + } + const char *h = q + strlen("://"); + size_t hl = strcspn(h, "/ \t\r\n\"'"); + for (size_t i = hl; i > 0; i--) { + if (h[i - SKIP_ONE] == '@') { + hl -= i; + h += i; + break; + } + } + size_t colon = strcspn(h, ":"); + return env_copy(host, sz, h, colon < hl ? colon : hl); +} + +/* `host:port` with a 2-5 digit port. */ +static bool env_hostport(const char *v, char *host, size_t sz) { + enum { PORT_MIN = 2, PORT_MAX = 5 }; + const char *c = strrchr(v, ':'); + size_t pl = c ? strlen(c + SKIP_ONE) : 0; + if (!c || c == v || !isalnum((unsigned char)v[0]) || pl < PORT_MIN || pl > PORT_MAX || + strspn(c + SKIP_ONE, "0123456789") != pl) { + return false; + } + for (const char *p = v; p < c; p++) { + if (!isalnum((unsigned char)*p) && *p != '.' && *p != '-') { + return false; + } + } + return env_copy(host, sz, v, (size_t)(c - v)); +} + +/* A `..svc` name anywhere in the value. */ +static bool env_svc_dns(const char *v, char *host, size_t sz) { + for (const char *m = strstr(v, ".svc"); m; m = strstr(m + SKIP_ONE, ".svc")) { + char after = m[strlen(".svc")]; + if (isalnum((unsigned char)after) || after == '_') { + continue; + } + const char *ns = m; + while (ns > v && env_label_char(ns[-SKIP_ONE])) { + ns--; + } + if (ns == m || ns == v || ns[-SKIP_ONE] != '.' || ns[0] == '-') { + continue; + } + const char *name = ns - SKIP_ONE; + while (name > v && env_label_char(name[-SKIP_ONE])) { + name--; + } + bool word_start = + name == v || !(isalnum((unsigned char)name[-SKIP_ONE]) || name[-SKIP_ONE] == '_'); + if (name == ns - SKIP_ONE || name[0] == '-' || !word_start) { + continue; + } + return env_copy(host, sz, name, (size_t)(m + strlen(".svc") - name)); + } + return false; +} + +int cbm_env_address(const char *key, const char *value, char *svc, size_t svc_sz, char *ns, + size_t ns_sz) { + char host[CBM_SZ_256]; + if (!key || !value || !svc || !ns || svc_sz == 0 || ns_sz == 0) { + return 0; + } + bool found = env_addr_key(key) && env_bare_name(value) && + env_copy(host, sizeof(host), value, strlen(value)); + if (!found && (strstr(value, "{{") || strchr(value, '$'))) { + return 0; + } + found = found || env_url_host(value, host, sizeof(host)) || + env_hostport(value, host, sizeof(host)) || env_svc_dns(value, host, sizeof(host)); + if (!found) { + return 0; + } + int labels = 0; + const char *third = NULL; + for (const char *p = host; p;) { + const char *dot = strchr(p, '.'); + size_t len = dot ? (size_t)(dot - p) : strlen(p); + if (labels == 0 && !env_copy(svc, svc_sz, p, len)) { + return 0; + } + if (labels == SKIP_ONE && !env_copy(ns, ns_sz, p, len)) { + return 0; + } + if (labels == PAIR_LEN) { + third = p; + } + labels++; + p = dot ? dot + SKIP_ONE : NULL; + } + if (labels == SKIP_ONE) { + ns[0] = '\0'; + } + for (const char *p = svc; *p; p++) { + if (!env_label_char(*p)) { + return 0; + } + } + if (strcmp(svc, "localhost") == 0 || strspn(svc, "0123456789") == strlen(svc)) { + return 0; + } + bool svc_form = + labels <= PAIR_LEN || (third && strncmp(third, "svc", strlen("svc")) == 0 && + (third[strlen("svc")] == '\0' || third[strlen("svc")] == '.')); + return svc_form ? labels : 0; +} + int cbm_classify_string(const char *str, int len) { if (!str || len < PAIR_LEN) { return NOT_FOUND; diff --git a/internal/cbm/helpers.h b/internal/cbm/helpers.h index 724466679..f7c18c808 100644 --- a/internal/cbm/helpers.h +++ b/internal/cbm/helpers.h @@ -23,6 +23,17 @@ bool cbm_is_keyword(const char *name, CBMLanguage lang); // stay in sync with kPyBuiltinNodes in internal/cbm/lsp/py_builtins.c. bool cbm_is_resolvable_builtin(const char *name, CBMLanguage lang); +// The service an environment variable's value addresses -- never the value +// itself: the host of a URL, of `host:port`, a `..svc[.]` +// name inside it, or the whole value when `key` names an address (*_ADDR, +// *_ADDRESS, *_HOST, *_URL, *_ENDPOINT, *_URI) and it is a bare DNS name. A +// templated value, localhost or an IP addresses nothing. Writes the host's +// first label to `svc` and its second (`svc.ns`, `svc.ns.svc...`) to `ns` +// ("" for one label), and returns the label count when the host is a +// service DNS name (`svc`, `svc.ns`, `svc.ns.svc[...]`), else 0. +int cbm_env_address(const char *key, const char *value, char *svc, size_t svc_sz, char *ns, + size_t ns_sz); + // Classify a string literal as URL, config, or neither. // Returns CBM_STRREF_URL (0), CBM_STRREF_CONFIG (1), or -1 for neither. int cbm_classify_string(const char *str, int len); diff --git a/src/cli/cli.c b/src/cli/cli.c index 65902d4ca..6f15c7d47 100644 --- a/src/cli/cli.c +++ b/src/cli/cli.c @@ -1568,7 +1568,9 @@ static const char skill_content[] = "HANDLES, IMPLEMENTS, OVERRIDE, USAGE, CALL_REFERENCE, CONFIGURES, REFERENCES_FILE,\n" "MENTIONS (doc comment -> referenced code), FILE_CHANGES_WITH, SIMILAR_TO,\n" "SEMANTICALLY_RELATED, CONTAINS_FILE, CONTAINS_FOLDER,\n" - "INFRA_MAPS (K8s Service -> selected workloads), DEPENDS_ON (chart / manifest -> dependency)\n" + "INFRA_MAPS (K8s Service -> selected workloads), DEPENDS_ON (chart / manifest -> dependency),\n" + "DEPLOYS (what puts manifests / charts on a cluster -> what it puts there),\n" + "CONNECTS_TO (workload / compose service -> the Service its environment addresses)\n" "\n" "## Cypher Examples (for query_graph)\n" "```\n" diff --git a/src/mcp/mcp.c b/src/mcp/mcp.c index 03741e23a..7a62c8656 100644 --- a/src/mcp/mcp.c +++ b/src/mcp/mcp.c @@ -8726,10 +8726,11 @@ static yyjson_doc *resolve_trace_edge_types(const char *args, const char *mode, "CROSS_GRAPHQL_CALLS", "CROSS_TRPC_CALLS"}; /* topology: traffic (INFRA_MAPS), object references (USAGE), ordering * (DEPENDS_ON), controllers (HANDLES), what an object declares - * (DEFINES: an Ingress's routes) and what it deploys (DEPLOYS: a Flux - * Kustomization's path, a HelmRelease's chart) */ - static const char *mode_topology[] = {"INFRA_MAPS", "USAGE", "DEPENDS_ON", - "HANDLES", "DEFINES", "DEPLOYS"}; + * (DEFINES: an Ingress's routes), what it deploys (DEPLOYS: a Flux + * Kustomization's path, a HelmRelease's chart) and the services its + * containers' environment addresses (CONNECTS_TO) */ + static const char *mode_topology[] = {"INFRA_MAPS", "USAGE", "DEPENDS_ON", "HANDLES", + "DEFINES", "DEPLOYS", "CONNECTS_TO"}; /* deploy: what a CI job, script or Make rule starts (SPAWNS: scripts, * make targets, programs), calls (CALLS: shell functions) and puts on a * cluster (DEPLOYS) -- outbound from the job, inbound from a chart */ diff --git a/src/pipeline/pass_ci.c b/src/pipeline/pass_ci.c index c8b9a4467..2fc9e0912 100644 --- a/src/pipeline/pass_ci.c +++ b/src/pipeline/pass_ci.c @@ -27,6 +27,7 @@ #include "foundation/compat.h" #include "pipeline/pipeline_internal.h" #include "cbm.h" +#include "helpers.h" /* cbm_env_address */ #include "lang_specs.h" #include @@ -718,8 +719,13 @@ static const char *ci_unquote(const char *v, size_t *len) { return v; } +static const char *ci_interp(CBMArena *a, const char *s, const ci_env_t *env); + /* One .env line s[0, n): KEY=VALUE (an `export ` prefix, quotes around the - * value removed) appended to env; a comment, a blank or another line: none. */ + * value removed) appended to env, its ${X} references expanded from the lines + * before it as compose does (`CART_ADDR=cart:${CART_PORT}`; a single-quoted + * value stays literal, an unknown variable leaves it as written); a comment, + * a blank or another line: none. */ static void ci_dotenv_line(CBMArena *a, const char *s, size_t n, ci_env_t *env) { static const char EXPORT[] = "export "; size_t b = 0; @@ -743,9 +749,12 @@ static void ci_dotenv_line(CBMArena *a, const char *s, size_t n, ci_env_t *env) v++; } size_t vl = n - v; + bool literal = v < n && s[v] == '\''; const char *val = ci_unquote(s + v, &vl); + const char *raw = cbm_arena_strndup(a, val, vl); + const char *expanded = raw && !literal ? ci_interp(a, raw, env) : NULL; env->kv[PAIR_LEN * env->n] = cbm_arena_strndup(a, s + b, kl); - env->kv[PAIR_LEN * env->n + SKIP_ONE] = cbm_arena_strndup(a, val, vl); + env->kv[PAIR_LEN * env->n + SKIP_ONE] = expanded ? expanded : raw; env->n++; } @@ -988,35 +997,60 @@ static int ci_compose_deps(const ci_t *c, TSNode svc, const char ***out) { return n; } +/* The compose service a variable's value addresses (its .env interpolated): + * a bare service name only -- the value itself is never kept. NULL: none. */ +static const char *ci_env_host(const ci_t *c, const char *name, const char *raw, + const ci_env_t *env) { + enum { CI_DNS_LABEL = 64 }; + const char *value = raw ? ci_interp(c->a, raw, env) : NULL; + char svc[CI_DNS_LABEL]; + char ns[CI_DNS_LABEL]; + return value && cbm_env_address(name, value, svc, sizeof(svc), ns, sizeof(ns)) == SKIP_ONE + ? cbm_arena_strdup(c->a, svc) + : NULL; +} + /* The variable names a service's environment sets: a mapping's keys, a * list's "NAME=value" / "NAME" items (the name only -- a value is never - * kept), with their 1-based lines. */ -static int ci_compose_env(const ci_t *c, TSNode svc, const char ***out, const int **lines) { + * kept), with their 1-based lines and the service each value addresses. */ +static int ci_compose_env(const ci_t *c, TSNode svc, const ci_env_t *dotenv, const char ***out, + const int **lines, const char ***hosts) { int n = 0; TSNode env = ci_get(c, svc, "environment"); TSNode *items = ci_items(c, env, &n); ci_pair_t *pairs = items ? NULL : ci_pairs(c, env, &n); const char **names = cbm_arena_alloc(c->a, (size_t)(n + SKIP_ONE) * sizeof(*names)); int *rows = cbm_arena_alloc(c->a, (size_t)(n + SKIP_ONE) * sizeof(*rows)); + const char **to = cbm_arena_alloc(c->a, (size_t)(n + SKIP_ONE) * sizeof(*to)); int k = 0; - for (int i = 0; names && rows && i < n; i++) { + for (int i = 0; names && rows && to && i < n; i++) { int row = 0; const char *name = NULL; + const char *raw = NULL; if (items) { const char *s = ci_scalar(c, items[i], &row); const char *eq = s ? strchr(s, '=') : NULL; name = eq ? cbm_arena_strndup(c->a, s, (size_t)(eq - s)) : s; + raw = eq ? eq + SKIP_ONE : NULL; } else { + int vrow = 0; name = pairs[i].key; row = pairs[i].row; + raw = ts_node_is_null(pairs[i].value) ? NULL : ci_scalar(c, pairs[i].value, &vrow); + } + if (name && name[0] && !raw) { + /* a name without a value takes the project environment's (.env) */ + raw = cbm_arena_sprintf(c->a, "${%s}", name); } if (name && name[0]) { names[k] = name; + to[k] = ci_env_host(c, name, raw, dotenv); rows[k++] = row + SKIP_ONE; } } *out = names; *lines = rows; + *hosts = to; return k; } @@ -1044,7 +1078,7 @@ static void ci_compose(const ci_t *c, TSNode root, const char *dir, const ci_env .line = (int)ts_node_start_point(svc).row, .end_line = (int)ts_node_end_point(svc).row + SKIP_ONE}; s.ndeps = ci_compose_deps(c, svc, &s.deps); - s.nenv = ci_compose_env(c, svc, &s.env, &s.env_lines); + s.nenv = ci_compose_env(c, svc, env, &s.env, &s.env_lines, &s.env_hosts); c->service_fn(&s, c->ud); } } diff --git a/src/pipeline/pass_docker.c b/src/pipeline/pass_docker.c index b9cb65eba..b845211a4 100644 --- a/src/pipeline/pass_docker.c +++ b/src/pipeline/pass_docker.c @@ -1563,9 +1563,30 @@ static void dk_compose_edge(dk_compose_t *c, int64_t src, int64_t tgt, const cha } } +/* The services of the same file a service's environment addresses: + * CONNECTS_TO {via:env, key, container} -- the variable, never its value. */ +static void dk_compose_connects(dk_compose_t *c, const cbm_compose_service_t *s, int64_t self) { + for (int i = 0; s->env_hosts && i < s->nenv; i++) { + const char *hq = s->env_hosts[i] ? dk_compose_qn(c, s->env_hosts[i]) : NULL; + const cbm_gbuf_node_t *to = hq ? cbm_gbuf_find_by_qn(c->ctx->gbuf, hq) : NULL; + if (!to) { + continue; + } + char kesc[CBM_SZ_256]; + char sesc[CBM_SZ_256]; + cbm_json_escape(kesc, (int)sizeof(kesc), s->env[i]); + cbm_json_escape(sesc, (int)sizeof(sesc), s->name); + char props[CBM_SZ_1K]; + snprintf(props, sizeof(props), "{\"via\":\"env\",\"key\":\"%s\",\"container\":\"%s\"}", + kesc, sesc); + dk_compose_edge(c, self, to->id, "CONNECTS_TO", props); + } +} + /* A compose service: a Resource {ecosystem:compose, kind:Service} the file * DEFINES; it DEPENDS_ON the services it depends on / links, USAGE - * {ref:image} its image and DEPENDS_ON {via:build} the Dockerfile it builds. */ + * {ref:image} its image, DEPENDS_ON {via:build} the Dockerfile it builds and + * CONNECTS_TO the services its environment addresses. */ static void dk_compose_service(const cbm_compose_service_t *s, void *ud) { dk_compose_t *c = ud; const char *qn = dk_compose_qn(c, s->name); @@ -1600,6 +1621,7 @@ static void dk_compose_service(const cbm_compose_service_t *s, void *ud) { const cbm_gbuf_node_t *dep = dq ? cbm_gbuf_find_by_qn(c->ctx->gbuf, dq) : NULL; dk_compose_edge(c, self->id, dep ? dep->id : 0, "DEPENDS_ON", "{\"via\":\"depends_on\"}"); } + dk_compose_connects(c, s, self->id); if (s->image) { char esc[CBM_SZ_1K]; cbm_json_escape(esc, (int)sizeof(esc), s->image); diff --git a/src/pipeline/pass_k8s.c b/src/pipeline/pass_k8s.c index a3113a37b..26972efe8 100644 --- a/src/pipeline/pass_k8s.c +++ b/src/pipeline/pass_k8s.c @@ -1124,8 +1124,10 @@ typedef struct { char *ns; /* explicit target namespace, NULL = the source's */ char *name; /* target metadata.name */ const char *via; /* static: the extractor's table */ - const char *edge; /* static: "USAGE", "INFRA_MAPS" or "DEPENDS_ON" */ + const char *edge; /* static: "USAGE", "INFRA_MAPS", "DEPENDS_ON" or "CONNECTS_TO" */ char *route; /* an exposed route "" the edge leaves from, NULL = src */ + char *key; /* CONNECTS_TO: the environment variable addressing the target */ + char *container; /* CONNECTS_TO: its container, NULL = unnamed */ } k8s_ref_t; /* A CRD and the group/kind it declares. */ @@ -1198,6 +1200,8 @@ static void k8s_facts_free(k8s_facts_t *f) { cbm_free(CBM_MEM_CLASS_OTHER, f->refs[i].ns); cbm_free(CBM_MEM_CLASS_OTHER, f->refs[i].name); cbm_free(CBM_MEM_CLASS_OTHER, f->refs[i].route); + cbm_free(CBM_MEM_CLASS_OTHER, f->refs[i].key); + cbm_free(CBM_MEM_CLASS_OTHER, f->refs[i].container); } for (int i = 0; i < f->crd_n; i++) { cbm_free(CBM_MEM_CLASS_OTHER, f->crds[i].group); @@ -1331,7 +1335,11 @@ static void k8s_facts_keep(k8s_facts_t *f, const CBMFileResult *res, const int64 .name = x->name ? cbm_mem_strdup(CBM_MEM_CLASS_OTHER, x->name) : NULL, .via = x->via, .edge = x->edge, - .route = route ? cbm_mem_strdup(CBM_MEM_CLASS_OTHER, x->route) : NULL}; + .route = route ? cbm_mem_strdup(CBM_MEM_CLASS_OTHER, x->route) : NULL, + .key = x->key ? cbm_mem_strdup(CBM_MEM_CLASS_OTHER, x->key) : NULL, + .container = x->key && x->container + ? cbm_mem_strdup(CBM_MEM_CLASS_OTHER, x->container) + : NULL}; } } } @@ -1888,12 +1896,21 @@ static void k8s_resolve_refs(cbm_pipeline_ctx_t *ctx, k8s_bundles_t *b, const k8 } } atomic_fetch_add_explicit(&g_k8s_candidates, (uint64_t)(n), memory_order_relaxed); - char props[CBM_SZ_256]; - snprintf(props, sizeof(props), "{\"%s\":\"%s\"}", - strcmp(ref->edge, "INFRA_MAPS") == 0 ? "kind" - : strcmp(ref->edge, "DEPENDS_ON") == 0 ? "via" - : "ref", - ref->via); + char props[CBM_SZ_1K]; + if (ref->key) { /* CONNECTS_TO: the variable and container, never the value */ + char kesc[CBM_SZ_256]; + char cesc[CBM_SZ_256]; + cbm_json_escape(kesc, (int)sizeof(kesc), ref->key); + cbm_json_escape(cesc, (int)sizeof(cesc), ref->container ? ref->container : "-"); + snprintf(props, sizeof(props), "{\"via\":\"%s\",\"key\":\"%s\",\"container\":\"%s\"}", + ref->via, kesc, cesc); + } else { + snprintf(props, sizeof(props), "{\"%s\":\"%s\"}", + strcmp(ref->edge, "INFRA_MAPS") == 0 ? "kind" + : strcmp(ref->edge, "DEPENDS_ON") == 0 ? "via" + : "ref", + ref->via); + } /* A name is one object per kind and namespace, so two declarations * of it in one bundle are alternatives (versioned manifests side by * side): the declaration in the reference's own file, when there is diff --git a/src/pipeline/pipeline_internal.h b/src/pipeline/pipeline_internal.h index 8985e24e2..b956e4293 100644 --- a/src/pipeline/pipeline_internal.h +++ b/src/pipeline/pipeline_internal.h @@ -1265,7 +1265,8 @@ typedef void (*cbm_build_site_fn)(const cbm_build_site_t *site, void *ud); * (repository path; NULL: no build, or unknown), the service names its * depends_on and links list, and the names of the environment variables its * environment sets (map keys / list items' names -- never a value), with - * their 1-based lines. */ + * their 1-based lines and the service name each value addresses (a bare + * name, `name:port`, a URL's host; NULL: none -- cbm_env_address). */ typedef struct { const char *name; const char *image; @@ -1274,6 +1275,7 @@ typedef struct { int ndeps; const char **env; const int *env_lines; + const char **env_hosts; int nenv; int line; int end_line; @@ -1490,8 +1492,9 @@ int cbm_pipeline_build_fresh_semantic_manifest(cbm_pipeline_t *p, const char *pr * files the container's image is built from (pass_envjoin.c); a K8s * image reference says its container (USAGE {ref:image, container}) and * a Dockerfile's Module its image's files (image_context, image_files, - * image_ignore). An index built before holds none of them, so it - * rebuilds once on upgrade. */ + * image_ignore). A workload or compose service CONNECTS_TO the Service + * its environment addresses (extract_k8s.c, pass_docker.c). An index + * built before holds none of them, so it rebuilds once on upgrade. */ enum { CBM_SEMANTIC_INDEX_VERSION = 9 }; typedef struct { diff --git a/tests/test_grammar_probe_f.c b/tests/test_grammar_probe_f.c index e9e92a702..02873d0bf 100644 --- a/tests/test_grammar_probe_f.c +++ b/tests/test_grammar_probe_f.c @@ -2492,6 +2492,95 @@ static char *gpf_env_joins(const GpfFile *files, int nfiles) { return out; } +/* Every CONNECTS_TO edge as " -> ()". */ +static char *gpf_connects(const GpfFile *files, int nfiles) { + GpfProj lp; + cbm_store_t *store = gpf_index_files(&lp, files, nfiles); + char *out = NULL; + if (store) { + sqlite3_stmt *st = NULL; + const char *sql = + "SELECT coalesce((SELECT group_concat(x, char(10)) FROM (SELECT s.file_path || ' ' || " + "s.name || ' -> ' || t.name || ' (' || json_extract(e.properties,'$.key') || ')' AS x " + "FROM edges e JOIN nodes s ON s.id=e.source_id JOIN nodes t ON t.id=e.target_id WHERE " + "e.project=?1 AND e.type='CONNECTS_TO' ORDER BY x)), '')"; + if (sqlite3_prepare_v2(cbm_store_get_db(store), sql, -1, &st, NULL) == SQLITE_OK) { + sqlite3_bind_text(st, 1, lp.project, -1, SQLITE_STATIC); + if (sqlite3_step(st) == SQLITE_ROW) { + out = strdup((const char *)sqlite3_column_text(st, 0)); + } + } + sqlite3_finalize(st); + } + gpf_cleanup(&lp, store); + return out; +} + +/* A service address in a container's environment CONNECTS_TO the Service it + * names: `svc:port`, a URL's host, `..svc...`, a bare name on an + * address key -- a K8s Service of the bundle in the named (else the + * object's own) namespace, a compose service of the same file, its .env + * interpolated (a name without a value takes the .env's, which may itself + * name others). Never: another domain's host, a templated value, localhost, + * a bare name on any other key, a namespace without that Service, a service + * addressing itself -- and never a value in the graph. */ +TEST(probe_env_address_connects_to_its_service) { + static const GpfFile files[] = { + {"k8s/app.yaml", + "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: web\n namespace: shop\n" + "spec:\n template:\n spec:\n containers:\n - name: web\n" + " image: acme/web\n env:\n" + " - name: CATALOG_ADDR\n value: \"catalog:3550\"\n" + " - name: AUTH_URL\n" + " value: \"http://auth.auth.svc.cluster.local:8080/login\"\n" + " - name: CACHE_HOST\n value: cache\n" + " - name: DB_HOST\n value: db.example.com\n" + " - name: TEMPLATED_ADDR\n value: \"{{ .Values.addr }}\"\n" + " - name: LOCAL_ADDR\n value: \"localhost:9000\"\n" + " - name: GREETING\n value: catalog\n" + " - name: PRICES_ADDR\n value: \"catalog.other:3550\"\n" + " - name: MAIL_URL\n value: \"smtp://mail.shop.example.com:25\"\n"}, + {"k8s/svc.yaml", "apiVersion: v1\nkind: Service\nmetadata:\n name: catalog\n" + " namespace: shop\n---\napiVersion: v1\nkind: Service\nmetadata:\n" + " name: cache\n namespace: shop\n---\napiVersion: v1\nkind: Service\n" + "metadata:\n name: auth\n namespace: auth\n---\napiVersion: v1\n" + "kind: Service\nmetadata:\n name: db\n namespace: shop\n---\n" + "apiVersion: v1\nkind: Service\nmetadata:\n name: mail\n" + " namespace: shop\n"}, + {"compose.yaml", + "services:\n web:\n image: acme/web\n environment:\n DB_HOST: db\n" + " CACHE_URL: redis://cache:6379\n API_KEY: s3cr3t-value\n" + " WEB_ADDR: web\n worker:\n image: acme/worker\n environment:\n" + " - QUEUE_ADDR\n - OTHER_URL=http://${QUEUE_HOST}:80\n" + " db:\n image: postgres\n cache:\n image: redis\n queue:\n" + " image: rabbitmq\n"}, + {".env", "QUEUE_PORT=5672\nQUEUE_HOST=queue\nQUEUE_ADDR=queue:${QUEUE_PORT}\n"}}; + int n = (int)(sizeof(files) / sizeof(files[0])); + char *got = gpf_connects(files, n); + static const char want[] = "compose.yaml Service/web -> Service/cache (CACHE_URL)\n" + "compose.yaml Service/web -> Service/db (DB_HOST)\n" + "compose.yaml Service/worker -> Service/queue (QUEUE_ADDR)\n" + "k8s/app.yaml Deployment/web -> Service/auth (AUTH_URL)\n" + "k8s/app.yaml Deployment/web -> Service/cache (CACHE_HOST)\n" + "k8s/app.yaml Deployment/web -> Service/catalog (CATALOG_ADDR)"; + if (!got || strcmp(got, want) != 0) { + fprintf(stderr, "connects got:\n%s\nwant:\n%s\n", got ? got : "(null)", want); + free(got); + FAIL("CONNECTS_TO edge set"); + } + free(got); + static const char *const values[] = {"catalog:3550", "s3cr3t-value", "redis://cache:6379", + "queue:5672", "/login"}; + for (size_t i = 0; i < sizeof(values) / sizeof(values[0]); i++) { + int hits = gpf_graph_mentions(files, n, values[i]); + if (hits != 0) { + fprintf(stderr, "value %s: %d graph mentions\n", values[i], hits); + FAIL("an env value reached the graph"); + } + } + PASS(); +} + /* A variable a container sets joins the readers of its name inside the files * its own image is built from: the K8s container's image (two containers, * two images), built by compose from a Dockerfile; the stages the last one @@ -4823,6 +4912,7 @@ SUITE(grammar_probe_f) { RUN_TEST(probe_shell_docker_builds_build_images); RUN_TEST(probe_container_env_sets_scoped_envvars); RUN_TEST(probe_container_env_joins_its_own_image_readers); + RUN_TEST(probe_env_address_connects_to_its_service); RUN_TEST(probe_image_hub_splits_by_component); RUN_TEST(probe_gitops_objects_deploy_their_units); RUN_TEST(probe_shell_kubectl_deploys_what_it_applies);