b
Senior Big Data Engineer (Python | AWS)
beo software private limited(Other)
Job Publish Date: 19 hours ago
India, Cochin / Kochi / ErnakulamFull TimeFresherWork From OfficeSource : Foundit
Job Description
Responsibilities
Role Summary
Own the data platform that powers clinician discovery, credential unification, and care-quality analytics. Design resilient, low-latency ingestion and transformation at scale (batch + streaming) with GDPR-by-design. Your work underpins search, matching, and ML features in our telemedicine platform across Germany.
Key Responsibilities
Desired Candidate Profile
How We Work & Benefits
Back
Role Summary
Own the data platform that powers clinician discovery, credential unification, and care-quality analytics. Design resilient, low-latency ingestion and transformation at scale (batch + streaming) with GDPR-by-design. Your work underpins search, matching, and ML features in our telemedicine platform across Germany.
Key Responsibilities
- Design and operate AWS-native data lakehouse: Amazon S3 + Lake Formation (governance),
- Build high-throughput ingestion and CDC pipelines from partner APIs, files, and databases
- Implement idempotent upserts, deduplication, and delta detection; define source-of-truth
- Model healthcare provider data (DDD) and normalize structured/semi-structured payloads
- Engineer vector-aware datasets for clinician/patient matching; operate pgvector on Amazon.
- Establish data quality (freshness, accuracy, coverage, cost-per-item) with automated checks.
- Harden security & privacy: IAM least-privilege, KMS encryption, Secrets Manager, VPC
- Observability-first pipelines using OpenTelemetry (ADOT), CloudWatch, X-Ray; DLQ
- Performance tuning for Aurora PostgreSQL (incl. indexing, partitioning, vacuum/analyze)
- CI/CD for data (Terraform/CDK, GitHub Actions/CodeBuild/CodePipeline); test automation
Desired Candidate Profile
- 6+ years in data engineering at scale; proven delivery in production systems (regulated
- Expertise in Python and SQL; hands-on with Spark (EMR/Glue) and stream processing
- Deep AWS experience across S3, Glue, Athena, Redshift or Aurora PostgreSQL, Lake
- PostgreSQL mastery incl. query planning, indexing, and performance tuning; familiarity with
- Strong grasp of idempotency, deduplication, CDC, schema evolution, SCDs, and contract
- Observability (OpenTelemetry), CI/CD, and IaC (Terraform/CDK) best practices; strong
- Security-by-design mindset: data minimization, encryption, secrets, PII-safe logging; working knowledge of GDPR and auditability.
- Effective communicator across Product, Platform, Data Science, and Compliance; pragmatic,metrics-driven delivery.
- Experience with FHIR/HL7, German TI/ePrescription/ePA integrations.
- DBT for transformations; OpenMetadata/Amundsen for catalog/lineage.
- Go for high-throughput services; experience with Bedrock or SageMaker for embedding
How We Work & Benefits
- API-first, clean architecture, and pairing culture; mandatory code reviews.
- Remote-friendly with defined core hours; mission-led, patient-safety-first.
- Ownership mindset: you build it, you run it (with sensible SLOs and error budgets).
- All PHI/PII processed within EU regions (e.g., eu-central-1); strict key management via AWS
- Right-to-erasure and lawful-basis handling embedded in data lifecycle (tombstones, purge
Back
Maximize your interview chances
Create tailored professional resumes in minutes using AI