| Metadata | Value |
|---|---|
| TAM Boundary | #10 of 12 |
| Mission | Data platform, analytics |
| Key Capabilities | Real-time streaming, data warehouse, business intelligence, ETL/ELT |
| Last Scanned | 2026-01-09 |
The Data boundary encompasses the Data Integration Platform (DIP) - a modern streaming data architecture that ingests events from operational systems via Kafka/MSK and processes them through Apache Flink workers. The platform enables near real-time data flow from transactional systems to Snowflake for analytics and reporting via PowerBI.
| Repo | Language | Application | Description |
|---|---|---|---|
| dataeng-dip-etl-template | Python | DIP | Template for Flink pipelines |
| dataeng-dip-etl-dt-auth-requestcreated | Python | DIP | Datatrans auth request created |
| dataeng-dip-etl-dt-auth-requestresponded | Python | DIP | Datatrans auth request responded |
| dataeng-dip-etl-dt-merchant | Python | DIP | Datatrans merchant data |
| dataeng-dip-etl-ife-auth-requestcreated | Python | DIP | IntegraFE request created |
| dataeng-dip-etl-ife-auth-requestresponded | Python | DIP | IntegraFE request responded |
| dataeng-dip-etl-nas-auth-acquirerrequested | Python | DIP | NAS acquirer requested |
| dataeng-dip-etl-nas-auth-acquirerresponded | Python | DIP | NAS acquirer responded |
| dataeng-dip-etl-psw-schemeresponded | Python | DIP | PSW scheme responded |
| dataeng-dip-etl-fibre-funding | Python | DIP | FIBRE funding data |
| dataeng-dip-etl-ccc-terminal-management | Python | DIP | Terminal management data |
| dataeng-dip-etl-bin-lookedup | Python | DIP | BIN lookup data |
| dataeng-dip-etl-enrichment-data-sink | Python | DIP | Enrichment data persistence |
| dataeng-dip-helm-charts | - | DIP | Helm charts for Flink deployments |
| dip-charts | Smarty | DIP | DIP Helm templates |
| data-snowflakeobjects | PLpgSQL | Snowflake | Non-dbt Snowflake objects |
| data-dbt | TSQL | Snowflake | DBT transformations |
| dbt_comm | Python | Snowflake | DBT common modules |
| pp-snowflake-int | Smarty | Snowflake | Planet Portal Snowflake integration |
| pp-reports-api | C# | Reporting | Payment reports API |
| mas-data-warehouse | Perl | MAS | MAS transform/load/report processes |
| planet-kafka-connect | Python | Kafka | Kafka-Connect cluster deployment |
Source Systems Kafka/MSK Flink Workers Snowflake
┌─────────────┐ ┌─────────┐ ┌───────────┐ ┌─────────┐
│ Datatrans │ ──────────────▶ │ │ ──────────▶ │ PyFlink │ ──────────▶ │ RAW │
│ Integra FE │ │ Topics │ │ Transform │ │ Curated │
│ NAS / PSW │ │ │ │ Workers │ │ Marts │
│ FIBRE │ └─────────┘ └───────────┘ └────┬────┘
└─────────────┘ │
▼
┌─────────┐
│ PowerBI │
└─────────┘
| Source | Topics | Flink Workers |
|---|---|---|
| Datatrans | dt.auth.* | dt-auth-requestcreated, dt-auth-requestresponded |
| Integra FE | ife.auth.* | ife-auth-requestcreated, ife-auth-requestresponded |
| NAS | nas.auth.* | nas-auth-acquirerrequested, nas-auth-acquirerresponded |
| PSW | psw.* | psw-schemeresponded |
| FIBRE | fibre.* | fibre-funding |
| Layer | Technology |
|---|---|
| Message Broker | Apache Kafka (AWS MSK) |
| Stream Processing | Apache Flink (PyFlink) |
| Data Serialization | Protocol Buffers (Protobuf) |
| CDC | Debezium |
| Data Warehouse | Snowflake |
| Transformations | dbt |
| Orchestration | ArgoCD, GitHub Actions |
| Container Runtime | Kubernetes (EKS) |
| IaC | Helm, Terraform |
| Schema | Purpose |
|---|---|
| Authorisation | Payment authorization events |
| Merchant | Merchant master data |
| Clearing | Transaction clearing events |
| Submission | Submission events |
| Funding | Merchant funding events |
| Terminal | Terminal management data |
| BIN | Card BIN lookup data |