Data Engineer | Databricks Certified | Process Automation Consultant | ETL/ELT, Cloud & Big Data

projects

github

Tom C.Rojo  • Peerlist

<p><strong>GDPR-Compliant Healthcare Data Pipeline</strong> is an AWS-native solution designed to process over 10 million daily health records while ensuring compliance with GDPR Article 17, also known as the Right to Erasure. This robust data pipeline leverages a variety of AWS services to provide a secure, efficient, and scalable solution for healthcare data management.</p><h3>Key Features</h3><ul><li><strong>GDPR Compliance:</strong> Ensures data erasure within a 2-minute SLA using a Lambda-based handler and Athena CTAS.</li><li><strong>Privacy by Design:</strong> Implements salted SHA-256 pseudonymization to protect sensitive data.</li><li><strong>Security:</strong> Utilizes customer-managed KMS keys, VPC endpoints, and zero public internet traversal for enhanced security.</li><li><strong>Infrastructure as Code:</strong> Deploys using CloudFormation for consistent and repeatable infrastructure management.</li><li><strong>Full Audit Trail:</strong> Maintains comprehensive logs in DynamoDB and CloudWatch for auditing purposes.</li></ul><h3>Architecture</h3><ul><li><strong>Ingestion:</strong> Data is ingested through Kinesis Firehose and stored in S3 as raw data.</li><li><strong>Processing:</strong> AWS Glue ETL processes data, storing curated results in S3 and Redshift Serverless.</li><li><strong>Compliance:</strong> A Lambda-based erasure handler ensures compliance with GDPR requirements.</li></ul><h3>Performance Benchmarks</h3><p>The pipeline achieves an ETL throughput of over 8,500 records per second at production scale, with a unit cost of $0.00005 per record. Monthly costs range from $3,000 to $4,200 for production environments.</p>