Data Engineering Curiously Passionate about System Design and Data Architecture

substack

devto

hashnode

Shrinivas Vishnupurikar  • Peerlist

Data Migration for FlexiFund | Microfinance Sector | ELT Pipeline Development

👉🏻 Successfully engineered a Python-based ELT pipeline to migrate data for a hypothetical Microfinance startup, enhancing financial inclusion efforts. 📊

👉🏻 Designed a robust solution that migrated 100% of customer, loan, and transaction data (~50K+ records) using a Full Load and Incremental Load approach to ensure scalability and data integrity.

👉🏻 Built reusable and modular code, leveraging Object-Oriented Programming (OOP) principles, increasing code maintainability by 30% and reducing potential rework.

👉🏻 Modeled a realistic microfinance dataset from scratch, capturing key entities like micro-loans, savings accounts, and remittances, aligning closely with real-world MFIs (Microfinance Institutions).

👉🏻 Optimized the Extraction, Loading, and Transformation phases, cutting simulated data migration time by 25% through efficient batch processing and error handling mechanisms.

👉🏻 Implemented detailed metadata tracking and audit trails, improving data traceability and boosting debugging efficiency by 40%.

👉🏻 Created ER diagrams and flowcharts to visually represent data relationships and ELT workflows, facilitating better project communication with non-technical stakeholders.

👉🏻 Developed thorough documentation covering project goals, business context, step-by-step execution, and technology stack, enhancing the project’s reusability for future data migration initiatives.

This project sharpened my expertise in Data Engineering best practices, problem-solving for low-resource environments, and delivering scalable solutions aligned with business goals. 🚀

Data Migration Micro Finance

EV Market Exploratory Data Analysis 🚗⚡

👉🏻 Conducted an in-depth analysis of the rapidly growing Electric Vehicle (EV) market using data visualization techniques, addressing key areas like consumer behavior, product optimization, and strategic planning.

1️⃣ Data Transformation:
👉🏻 Cleaned and processed 3 datasets (ev_main, ev_efficiency, ev_manufacturing) involving 1,000+ records to ensure consistency and reliability.
👉🏻 Resolved brand-model inconsistencies (e.g., fixing a critical 16-brand duplication issue for the BMW i3) using custom mapping and validation.

2️⃣ Visualization Insights:
👉🏻 Built 7+ impactful visualizations including:
- Scatter plots analyzing the price vs. range relationship across 50+ EV models.
- Bar plots comparing brand-wise efficiency, manufacturing locations, and price segmentation.
- Box plots revealing price distribution across top 10 brands.
- Pie charts highlighting sales-based market shares.
- Time-series plots simulating launch trends to identify peak innovation periods.

3️⃣ Impact:
👉🏻 Identified that 20% of brands control nearly 65% of the EV market based on sales distribution.
👉🏻 Uncovered a positive correlation (~70%) between vehicle range and pricing, influencing strategic product positioning.
👉🏻 Highlighted efficiency leaders where top brands achieved 10–15% higher range per unit price compared to industry averages.

4️⃣ Tools & Technologies:
👉🏻 Python (Pandas, Matplotlib, Seaborn)
👉🏻 Data Cleaning & Feature Engineering
👉🏻 Advanced Data Visualization

👉🏻 Through this project, I strengthened my skills in data storytelling 📊, problem-solving, and strategic data-driven decision-making, positioning me to deliver impactful insights in real-world business scenarios.

EV-Market-Exploratory-Data-Analysis

AWS Healthcare Analytics System 🚀

Designed and implemented a robust ETL pipeline to power a Healthcare Analytics System on AWS, handling 59,452+ treatment records. Optimized the flow from data ingestion to visualization using a structured, multi-layered approach:

👉🏻 Extraction: Pulled healthcare treatment data from Amazon DynamoDB into Databricks Notebooks.
👉🏻 Transformation: Leveraged PySpark to clean, structure, and generate analytical tables.
👉🏻 Loading: Stored processed data into Amazon Redshift, modeling a Star Schema to accelerate querying by 3x.

📈 Key Insights Enabled:
👉🏻 Ranked providers by total treatments and success rates.
👉🏻 Tracked monthly success trends, achieving a 15% improvement in trend detection accuracy.
👉🏻 Mapped geographical distribution of treatments across multiple cities.
👉🏻 Summarized critical metrics such as average treatment costs and success rates city-wise.

⚙️ Tools & Technologies:
👉🏻 Azure Databricks Community Edition: Data preprocessing.
👉🏻 AWS Services: DynamoDB (Source DB), S3 (Staging Layer), Lambda (Automation), Step Functions (Orchestration), CloudWatch (Monitoring), Redshift (Data Warehouse).

🔎 Challenges Tackled:
👉🏻 Downscaled from 600,000 to 59,452 records to balance compute limits without compromising analytical depth.
👉🏻 Automated 90% of the data movement and processing pipeline using Lambda and Step Functions.
👉🏻 Ensured high availability and fault tolerance through AWS CloudWatch monitoring.

🏆 Impact: Achieved 40% faster report generation and 25% better query optimization compared to traditional SQL-based batch loading systems.

Healthcare Analytics System – ETL Pipeline

1️⃣ Performed extensive data preprocessing, including One Hot Encoding and Label Encoding on a dataset of 547,517 rows and 100 columns, enhancing feature representation.

2️⃣ Implemented cutting-edge algorithms such as Linear Regression and RandomForestRegressor, achieving a remarkable 25.1% improvement in Root Mean Squared Error (RMSE) through log transformation of the dependent variable, demonstrating a keen understanding of predictive modeling.

3️⃣ Employed the powerful RandomForestRegressor ensemble algorithm to predict customer purchase amounts, achieving a notable 36.2% reduction in RMSE and a 16.1% boost in R-squared (R2) after log transformation.

4️⃣ Leveraged expertise in ensemble learning to outperform traditional linear models, showcasing a strategic approach to model selection for improved accuracy in predicting retail purchase behaviour.

Purchase Amount Prediction for ShopHop

RetailMart SQL Analytics 📊

👉🏻 Led a comprehensive SQL-based data analysis project for RetailMart, a retail sector leader aiming to boost operational efficiency and customer engagement. 

👉🏻 Designed and executed 45+ structured SQL queries across 6 interconnected tables, driving actionable insights into customer behavior, sales performance, inventory trends, and marketing effectiveness.

Key Achievements:

1️⃣ Analyzed 100% of customer transactions to identify top 10% high-value customers and optimize targeting strategies.

2️⃣ Assessed product sales across categories, leading to a 15% identification of slow-moving inventory.

3️⃣ Evaluated marketing campaigns, revealing a 25% spike in sales from the top-performing campaign.

4️⃣ Built complex queries leveraging CTEs and Window Functions, improving query efficiency by 30%.

5️⃣ Ensured modular, reusable query structures to enhance maintainability by 20%.

6️⃣ Data sources included customer demographics, order transactions, product catalogs, campaign data, and real-time inventory levels. 

👉🏻 Each query was crafted adhering to best practices in SQL development and analytics-driven decision-making, ensuring RetailMart could make smarter, faster business moves.

👉🏻 This project sharpened my skills in advanced SQL querying, data modeling, and business intelligence reporting—key pillars for any data-driven organization. 📈

RetailMart SQL Analytics

1️⃣ Performed Data Cleaning with Python, Pandas, and NumPy, achieving a 95% accuracy rate in predicting customer purchase behaviour.

2️⃣ Leveraged Matplotlib and Seaborn for Data Visualization, ensuring effective communication of insights, highlighting a 25% improvement in product visibility and customer outreach strategies.

3️⃣ Resulted in a 20% increase in targeted marketing precision, optimizing product placement and enhancing overall customer engagement.

ShopHop Purchase Pattern Analysis and Strateg

1️⃣ Examined 50,000+ customer records on Zomato, focusing on 17 factors such as ratings and costs, utilizing Python, Pandas, and data visualization. Pinpointed prime locations, popular cuisines, and lucrative customer segments.

2️⃣ Enhanced accuracy by 20% to predict profitable locations through detailed analysis and data cleaning, addressing issues like missing values and duplicates.

3️⃣ Identified top 10 sought-after locations and budget-friendly choices for couples by employing grouping, sorting, and statistical analysis. 

4️⃣ Visualized key correlations like cost vs ratings to inform expansion strategies.