IIIT Raichur CS | Agentic Systems Specialist (JARVIS OS) | Top 0.5% Amazon ML Challenge | Full-stack Engineer

leetcode

github

Soham Umbare  • Peerlist

This project features the implementation of a custom CLIP (Contrastive Language-Image Pretraining) model, leveraging a Vision Transformer (ViT) backbone and trained on the COCO dataset. Inspired by the original CLIP paper and Stable Diffusion's architecture, we developed a custom training pipeline that optimizes performance despite hardware limitations. Key Highlights: - Final loss: 0.72 after 85 epochs with a batch size of 1024. - Resource Efficiency: Trained on an NVIDIA Tesla P100 GPU (16 GiB), with 150 GPU hours of training time. - Custom training pipeline with efficient data loading and evaluation. - Zero-shot capabilities for tasks like image retrieval and classification. This project highlights expertise in transformer-based architectures, contrastive learning, and bridging vision-language modalities. Scripts for evaluating pre-trained models on custom inputs are available.

CLIP Model - Reimplementation, Training and E

Developed an Image Captioning model using PyTorch, integrating ResNet-50 for image feature extraction and an LSTM-based decoder for sequence generation. Achieved BLEU-1 and BLEU-2 scores of 68.4 and 53.3 on the Flickr 8k dataset. Adapted from a TensorFlow-based implementation by Hackers Realm, I reworked it for PyTorch with ResNet-50 and introduced significant training and data pipeline improvements, enhancing performance. Key Features: - ResNet-50 for detailed feature extraction. - LSTM-based decoder for word-by-word caption generation. - Modular design for preprocessing, training, evaluation, and inference.

Image Captioning with Pytorch

Developed a software with consists of a web app to track and manage prisoners along with staff and a flutter app to get current location of the prisoners

Prisoner Management System with Live Tracking

An AI-powered tool designed to create personalized academic and career roadmaps based on student information. The project utilizes a Retrieval-Augmented Generation (RAG) approach with the Mistral-7B model, featuring: Key Features: - User-Friendly Interface: Intuitive GUI for collecting detailed user data. - Personalized Recommendations: Tailored roadmaps aligned with global events, workshops, and academic goals. - Career and Academic Planning: Helps students identify skill-building opportunities and manage time effectively. - Mental Health Focus: Roadmaps prioritize a balanced approach, ensuring students maintain good mental health alongside their academic and career pursuits. - Current Events Integration: Generates roadmaps based on up-to-date events, scholarships, competitions, and opportunities worldwide. - AI Integration: Incorporates advanced language models for generating actionable insights. This project was developed collaboratively with Soham Bit, this project highlights the potential of AI in personalized education, career guidance, and holistic student development.

Roadmap-MakerAI

This project explores semantic segmentation of runway areas from aerial images to enable automated landing of rotorcraft on landing platforms. We experiment with a range of models—including both well-established architectures and novel attention-based designs—to evaluate their performance in real-world deployment conditions. 🎯 Objective To compare the efficiency and effectiveness of attention-based segmentation models (like MobileViT and Axial Attention) with traditional models (UNet, DeepLabV3+) for accurate and real-time runway detection. 🧪 Key Insights DeepLabV3+ with MobileNetV2 achieves the best accuracy-efficiency trade-off. Attention-based models (Axial, MobileViT) show potential but require more epochs and have higher test losses—suggesting room for optimization. MobileViT-UNet Lite, despite being the lightest, performed the low in accuracy but is suitable for low-latency environments. 🚀 Future Work Incorporating temporal consistency for videos using ConvLSTM/Transformer-based approaches. Deployment and benchmarking on real drone edge devices

🛬 Runway Segmentation for Autonomous Rotorcr

In this project we demonstrates an end-to-end machine learning workflow for predicting key properties (melting point and tensile strength) of alloys using composition–property data. 🎯 Objective: To develop and benchmark machine learning models capable of accurately predicting key mechanical properties—specifically melting point and tensile strength—from the chemical composition of high-temperature superalloys, with the ultimate goal of accelerating alloy design cycles, reducing experimental costs, and enabling data-driven discovery of novel compositions for aerospace and defense applications. 📋 Overview Materials informatics can accelerate alloy design by replacing costly experiments with rapid ML‑driven screening. In this project, I: 1. Curated a pilot dataset of ∼2,800 superalloys (composition + tensile strength + melting point). 2. Implemented and tuned: - K‑Nearest Neighbors (KNN) - Support Vector Regression (SVR) with linear, polynomial, and RBF kernels - Feed‑forward Artificial Neural Networks (ANNs), both single‑output and multi‑output 3. Compared performance using MSE and R² metrics.

The-Super-Alloys-Property-Prediction-by-ML

We developed a Variational Autoencoder (VAE) inspired by Stable Diffusion and the paper "Autoencoders by Dor Bank et al." Achieved near state-of-the-art performance on the COCO dataset (128x128 resolution) with MAE: 0.1031 and MSE: 0.03385. Key Features: - Custom loss function combining Perceptual Loss, KL Divergence Penalty, and Reconstruction Loss for improved semantic and structural fidelity. - Encoder-decoder architecture adapted from Stable Diffusion’s VAE, with novel modifications in training logic to enhance performance. - Trained over 600 epochs (8hrs) with stable convergence on NVIDIA Tesla P100 GPU. GitHub and Kaggle links available for implementation details.