jcoffey-dev
1de77b969f
ingest tags every record with a tenant_id Kafka header (built previously), but nothing consumed it to actually route the write. This closes that for ClickHouse: enterprise/cmd/enterprise-ingest (a second binary, mirroring enterprise-api) reuses ingest/consumer's own flush loop unchanged, with enterprise/internal/chwriter.Registry -- a per-tenant clickhousewriter.Writer registry -- swapped in as the writer. A batch pulled from the single shared Redpanda topic can mix records from many tenants, so WriteBatch groups by TenantID and dispatches each group to its own tenant's connection, fail- closed on an empty or unrecognized tenant_id. ingest/consumer and ingest/clickhousewriter move out of internal/ (same reason api/internal/* moved earlier this phase: enterprise/ can't import anything under another module's internal/). Their New() constructors now take small local Config structs instead of ingest/internal/config types, so enterprise/ doesn't need that import either. Building this surfaced a real bug: tenantprovision.ProvisionClickHouse only granted SELECT on a tenant's ClickHouse user, correct for chrunner's read-only use but not enough for chwriter reusing the same credential to write -- every real per-tenant write would have failed closed with a permission error. Fixed by widening the grant to SELECT, INSERT; no cross-tenant boundary is crossed by also allowing INSERT within a tenant's own database. Helm gates enterprise-ingest's Deployment on the same ingest.requireTenantCredential flag that already gates tag validation -- write-routing is meaningless without tagging already being required, so they're one decision, not two. docker-compose.yml's version is a disclosed, weaker approximation: it can't achieve Helm's genuine -mode=server/-mode=consumer split, so with the enterprise profile active both ingest and enterprise-ingest independently consume every message via different consumer groups -- harmless duplication for local verification only. Not built: Tantivy's independent Redpanda consumer (search/src/consumer.rs) still doesn't read the tenant_id header at all -- every record still lands in the one shared index regardless of tenant. Not run: the live-ClickHouse- gated tests (chwriter's cross-tenant routing test, tenantprovision's INSERT regression test) -- no Docker/database access in this environment; they're correct Go that has never executed, disclosed as such in docs/security/ threat-model.md and docs/phase-4-runbook.md §14.
109 lines
4.7 KiB
AMPL
109 lines
4.7 KiB
AMPL
module github.com/sentry/sentry/enterprise
|
|
|
|
go 1.25.0
|
|
|
|
// enterprise/ importing api/ (core) is the allowed direction of the
|
|
// module boundary hack/check-tenant-boundary.sh enforces -- see
|
|
// enterprise/internal/chrunner's doc comment: it implements api's
|
|
// executor.SQLRunner interface, which structurally requires importing
|
|
// the package that defines it.
|
|
replace github.com/sentry/sentry/api => ../api
|
|
|
|
// Same allowed direction, against ingest/ instead -- enterprise/internal/
|
|
// chwriter implements ingest/consumer's chWriter interface, which
|
|
// structurally requires importing the package that defines it (see
|
|
// that package's doc comment).
|
|
replace github.com/sentry/sentry/ingest => ../ingest
|
|
|
|
// api's own go.mod replace directive for proto/ is module-local and
|
|
// doesn't propagate here -- enterprise/ needs its own, or `go build`
|
|
// tries to fetch github.com/sentry/sentry/proto from a real (nonexistent)
|
|
// remote, since api/searchclient (now transitively imported) depends on
|
|
// the generated search gRPC stubs.
|
|
replace github.com/sentry/sentry/proto => ../proto
|
|
|
|
require (
|
|
github.com/sentry/sentry/api v0.0.0-00010101000000-000000000000
|
|
github.com/sentry/sentry/ingest v0.0.0-00010101000000-000000000000
|
|
)
|
|
|
|
require (
|
|
github.com/ClickHouse/clickhouse-go/v2 v2.48.0
|
|
github.com/coreos/go-oidc/v3 v3.20.0
|
|
github.com/crewjam/saml v0.5.1
|
|
github.com/go-jose/go-jose/v4 v4.1.4
|
|
github.com/google/uuid v1.6.0
|
|
github.com/jackc/pgx/v5 v5.10.0
|
|
github.com/sentry/sentry/proto v0.0.0-00010101000000-000000000000
|
|
golang.org/x/oauth2 v0.36.0
|
|
google.golang.org/grpc v1.83.0
|
|
k8s.io/api v0.31.0
|
|
k8s.io/apimachinery v0.31.0
|
|
k8s.io/client-go v0.31.0
|
|
)
|
|
|
|
require (
|
|
github.com/ClickHouse/ch-go v0.74.0 // indirect
|
|
github.com/andybalholm/brotli v1.2.2 // indirect
|
|
github.com/beevik/etree v1.5.0 // indirect
|
|
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
|
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
|
|
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
|
|
github.com/fxamacker/cbor/v2 v2.7.0 // indirect
|
|
github.com/go-faster/city v1.0.1 // indirect
|
|
github.com/go-faster/errors v0.7.1 // indirect
|
|
github.com/go-logr/logr v1.4.3 // indirect
|
|
github.com/go-openapi/jsonpointer v0.19.6 // indirect
|
|
github.com/go-openapi/jsonreference v0.20.2 // indirect
|
|
github.com/go-openapi/swag v0.22.4 // indirect
|
|
github.com/gogo/protobuf v1.3.2 // indirect
|
|
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
|
|
github.com/golang/protobuf v1.5.4 // indirect
|
|
github.com/google/gnostic-models v0.6.8 // indirect
|
|
github.com/google/go-cmp v0.7.0 // indirect
|
|
github.com/google/gofuzz v1.2.0 // indirect
|
|
github.com/imdario/mergo v0.3.6 // indirect
|
|
github.com/jackc/pgpassfile v1.0.0 // indirect
|
|
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
|
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
|
github.com/jonboulle/clockwork v0.2.2 // indirect
|
|
github.com/josharian/intern v1.0.0 // indirect
|
|
github.com/json-iterator/go v1.1.12 // indirect
|
|
github.com/klauspost/compress v1.19.1 // indirect
|
|
github.com/mailru/easyjson v0.7.7 // indirect
|
|
github.com/mattermost/xml-roundtrip-validator v0.1.0 // indirect
|
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
|
|
github.com/modern-go/reflect2 v1.0.2 // indirect
|
|
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
|
|
github.com/paulmach/orb v0.13.0 // indirect
|
|
github.com/pierrec/lz4/v4 v4.1.27 // indirect
|
|
github.com/pkg/errors v0.9.1 // indirect
|
|
github.com/russellhaering/goxmldsig v1.4.0 // indirect
|
|
github.com/segmentio/asm v1.2.1 // indirect
|
|
github.com/segmentio/kafka-go v0.4.51 // indirect
|
|
github.com/shopspring/decimal v1.4.0 // indirect
|
|
github.com/spf13/pflag v1.0.5 // indirect
|
|
github.com/x448/float16 v0.8.4 // indirect
|
|
go.opentelemetry.io/otel v1.44.0 // indirect
|
|
go.opentelemetry.io/otel/trace v1.44.0 // indirect
|
|
golang.org/x/crypto v0.54.0 // indirect
|
|
golang.org/x/net v0.57.0 // indirect
|
|
golang.org/x/sync v0.22.0 // indirect
|
|
golang.org/x/sys v0.47.0 // indirect
|
|
golang.org/x/term v0.45.0 // indirect
|
|
golang.org/x/text v0.40.0 // indirect
|
|
golang.org/x/time v0.3.0 // indirect
|
|
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
|
|
google.golang.org/protobuf v1.36.12 // indirect
|
|
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
|
|
gopkg.in/inf.v0 v0.9.1 // indirect
|
|
gopkg.in/yaml.v2 v2.4.0 // indirect
|
|
gopkg.in/yaml.v3 v3.0.1 // indirect
|
|
k8s.io/klog/v2 v2.130.1 // indirect
|
|
k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 // indirect
|
|
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8 // indirect
|
|
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
|
|
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect
|
|
sigs.k8s.io/yaml v1.4.0 // indirect
|
|
)
|