IIT Kharagpur

Ayush Debnath

B.Tech (Hons.) Mechanical Engineering + M.Tech Manufacturing Science & Engineering · Research intern at CMU & Stanford

I work on making neural networks efficient, physically grounded and trustworthy — benchmarking video world models for robotics, spiking networks for 3D perception, neural operators for PDEs, and multimodal models for clinical and field diagnosis.

Research openingsRésuméEmail meLinkedIn
World Models & Robot LearningSpiking Neural NetworksNeural Operators & PINNsVision–Language ModelsFederated LearningEfficient InferenceScientific Machine Learning
01

About

I'm a dual-degree student at IIT Kharagpur working across the boundary between machine learning and the physical sciences. Most of my research asks a version of the same question: how do we get neural networks to do more with less — less latency, less data, less private information leaving the device, or less disregard for the physics of the problem?

Right now that means evaluating video world models for robot learning at CMU's Robotics Institute, training-free SVG generation with diffusion and vision–language models in Zico Kolter's group, and neural PDE solvers for elastic wave propagation at Stanford. Earlier this year I built Adaptive Spiking PointNet at Purdue — an anytime 3D point-cloud classifier that decides for itself how many temporal slices it needs — and OmniMed-FL, a multimodal federated learning framework for clinical diagnosis, was accepted at IEEE GLOBECOM 2026. My bachelor thesis took the applied route: a vision–language pipeline for crop stress detection that had to survive contact with real farms.

Education
Dual Degree — B.Tech (Hons.) Mechanical Engineering + M.Tech Manufacturing Science & Engineering
Indian Institute of Technology Kharagpur · 2027
CGPA 8.03 / 10
All India Senior School Certificate Examination (Class XII)
South Point High School, Kolkata · 2021
95.8%
All India Secondary School Examination (Class X)
South Point High School, Kolkata · 2019
97%
02

Research & Experience

Research Intern · Carnegie Mellon University

Jul 2026 — Present

Mosam Dabhi, PhD · Prof. Simon Lucey · Prof. László Jeni — Robotics Institute

  • Running reproducible evaluations of video world models across WorldBench, WorldArena and Omni-WorldBench, identifying gaps in physical-state estimation, causal consistency and downstream-task performance.
  • Building Python pipelines that validate benchmark data, compare model results, generate publication-ready figures and tables, and keep the analysis deterministic through automated tests.
  • Designing evaluation protocols for state-aware robotic policies — benchmarking world models, vision-language-action models and diffusion policies across physical-reasoning and control tasks.
Video World ModelsRobot LearningVLA ModelsBenchmarking

Research Intern · Purdue University

Nov 2025 — Aug 2026

Prof. Sayeed Shafayet Chowdhury — Department of Computer Science

  • Developed Adaptive Spiking PointNet (ASP) for energy-efficient 3D point-cloud recognition on ModelNet40, built on LIF neurons and temporal spiking computation.
  • Designed an active slice-selection mechanism for anytime inference that processes only 2.4 of 16 temporal slices on average while maintaining strong classification performance.
  • Reached 89.10% on ModelNet40 and 93.28% on ModelNet10, outperforming fixed-order baselines at a 24.5% mean firing rate.
  • Implemented surrogate-gradient training, cross-attention-based slice selection and firing-rate regularization to tune the accuracy–latency–energy trade-off.
Spiking Neural Networks3D Point CloudsAnytime InferencePyTorch

Research Intern · Carnegie Mellon University

Oct 2025 — Present

Prof. Zico Kolter — School of Computer Science

  • Developed a training-free SVG generation and editing pipeline that uses pretrained diffusion models and frozen vision–language models to turn raster concepts into editable vector graphics.
  • Designed the DiffuSVG workflow: diffusion handles image-space generation, then a VLM extracts shapes, colours, positions and z-order to emit compact SVG markup.
  • Integrated CairoSVG, DiffVG, CLIP and DINO-based evaluation and refinement, enabling render–compare–refine iterations.
  • Achieved 92% valid SVG generation at roughly 3.5 elements per SVG, and documented the limitations on complex shapes.
Diffusion ModelsVision–Language ModelsVector GraphicsDiffVG

Research Intern · Stanford University

Dec 2025 — Present

Prof. Tapan Mukerji — Department of Energy Resources Engineering

  • Developing Physics-Informed Neural Networks (PINNs) and Fourier Neural Operators (FNOs) for elastic wave-propagation PDEs with governing physical constraints.
  • Benchmarking learning-based PDE solvers against finite-difference and finite-element methods on accuracy, stability and physical consistency.
  • Implemented physics-informed losses that combine governing-equation residuals with initial and boundary conditions to keep predictions physically valid.
  • Investigating FNO-based operator learning as a fast surrogate for wave dynamics across varying initial and boundary conditions.
PINNsFourier Neural OperatorsGeophysics

Bachelor Thesis Project · IIT Kharagpur

Apr 2025 — Jun 2026

Prof. Sudip Misra — Department of Computer Science and Engineering

  • Developed a multimodal crop-diagnosis framework combining VLMs, LLMs and agronomic metadata for automated crop-stress and disease assessment.
  • Designed a vision–language pipeline to extract semantic indicators — leaf discoloration, lesions, texture variations and other disease-related visual patterns.
  • Integrated an LLM reasoning module that fuses visual evidence with contextual and sensor information into interpretable diagnoses and actionable recommendations.
  • Built an end-to-end Flutter application on a Firebase backend for real-time image capture, inference and natural-language feedback.
VLMLLMFlutterFirebase

Computer Vision Intern · Indian Institute of Science, Bangalore

Dec 2024 — Apr 2025

Prof. S. N. Omkar — Aerospace Engineering Department

  • Implemented optical flow (Lucas–Kanade), YOLOv11, and MediaPipe for motion tracking and analysis.
  • Designed and implemented a novel motion-capture (MoCap) model from scratch using PyTorch, ResNet50 and CNNs.
  • Analyzed a dataset of over 20,000 images to extract keypoints in 3D human poses with six degrees of freedom.
  • Used generative models (GANs) to synthesize training images, reaching 0.80 accuracy on the augmented set.
Optical FlowYOLOv11ResNet50GANs

Full Stack Web Development Intern · IIT Roorkee

May 2024 — Jul 2024

Prof. Sudip Roy — Computer Science and Engineering Department

  • Created a dynamic website using HTML, CSS, JavaScript, React.js and Material-UI.
  • Implemented backend functionality with Node.js, Express.js and MongoDB for data management.
  • Added role-based login and a file storage system; deployed using AWS and Docker.
ReactNode.jsMongoDBAWSDocker

Machine Learning Intern · IIT Delhi

Jan 2024 — Apr 2024

Prof. Shaurya Shriyam — Mechanical Engineering Department

  • Developed an ANN-based model using displacement and direction changes for pedestrian trajectory prediction.
  • Improved trajectory accuracy through parameter tuning on 14,861 unidirectional and 9,673 bidirectional samples.
  • Analyzed Social-LSTM, Social GAN and SR-LSTM, integrating insights to enhance spatiotemporal accuracy.
ANNTrajectory PredictionLSTM
03

Selected Projects

OmniMed-FL — Multimodal Federated Learning for Clinical Diagnosis

Accepted at IEEE GLOBECOM 2026 (SAC E-Health). A federated framework that fuses medical imaging with clinical text while patient data stays decentralized.

Read the full write-up →
Details
  • Combines medical imaging and clinical text in a federated setup, so training never requires pooling patient data across sites.
  • Achieves 0.956 Macro-F1 under non-IID settings — 99.1% of the centralized model's performance.
  • Includes explainability and evidence retrieval so predictions can support, rather than replace, clinical decision-making.
Federated LearningMultimodalExplainabilityClinical AI

AI-Powered HR Chatbot — Open IIT OpenSoft 2025

Silver medal. An HR assistant that monitors employee well-being from conversations, flags at-risk employees, and explains why.

Details
  • Won Silver in Open IIT OpenSoft 2025 with a chatbot for employee engagement, well-being monitoring and personalised conversations.
  • Integrated GPT-4 for dialogue, DistilBERT sentiment analysis and SHAP-based explainability to identify at-risk employees and generate context-aware HR interactions.
  • Built the full stack on Next.js, FastAPI and PostgreSQL, with HR analytics dashboards, employee reports and automated workflows.
GPT-4DistilBERTSHAPNext.jsFastAPIPostgreSQL

Product Embeddings for Search Optimization

Scraped and embedded 14k Amazon products across 5 categories, then benchmarked embedding models with a contrastive scoring metric.

Details
  • Scraped 14k products across 5 categories from Amazon using BeautifulSoup4.
  • Plotted quantitative distributions in box plots after removing IQR outliers.
  • Used the TensorFlow Embedding Projector for 3D PCA and t-SNE visualizations of the word embeddings.
  • Developed a contrastive metric on which BERT outperformed Skip-Gram and GloVe by 10–15%.
BERTt-SNEBeautifulSoup4TensorFlow

Airline Delay Prediction

Delay forecasting over 3M flight records, paired with a genetic algorithm that reschedules flights to minimize propagated delay.

Details
  • Used XGBoost, ANN, LSTM and CNN-LSTM for delay prediction with temporal and categorical features.
  • Applied Pearson correlation, Kruskal–Wallis H-test, z-score, label encoding and IQR pruning on 3M records.
  • Implemented a genetic algorithm for flight rescheduling using selection, crossover and mutation.
  • Pulled data via API on AWS EC2 and added real-time weather with OpenCage; engineered season, traffic and time-of-day features.
XGBoostCNN-LSTMGenetic AlgorithmsAWS EC2

Mentorella — Mobile Application

A Flutter app connecting JEE aspirants with mentors in real time, built on Firebase.

Details
  • Contributed to Mentorella, a Flutter app enabling real-time mentor–mentee connections for JEE aspirants.
  • Integrated Firebase Authentication, Cloud Firestore and FCM for login, NoSQL storage and push notifications.
  • Developed session notification logic using Firestore queries, Cloud Functions and a custom REST API.
  • Used Provider and GoRouter, and optimized Firestore streams for better performance and lower backend usage.
FlutterFirebaseCloud FunctionsREST

Movie Platform

A MERN movie discovery platform with vector and fuzzy search across title, plot and cast.

Details
  • Built a scalable, responsive UI using React and Tailwind CSS.
  • Created the platform on the MERN stack with advanced language and genre filters.
  • Implemented vision search using OpenAI's model and MongoDB Atlas vector search.
  • Designed search by title, plot and cast using MongoDB fuzzy search, increasing search accuracy.
MERNMongoDB AtlasVector SearchTailwind
04

Awards

Silver — Open IIT OpenSoft 2025
AI-powered HR chatbot for employee well-being monitoring, built with GPT-4, DistilBERT, SHAP, Next.js, FastAPI and PostgreSQL.
JEE Advanced 2022 — AIR 4708
Among 250,000+ Mains-qualified candidates.
JEE Main 2022 — 99.1 percentile
Top 1% among over 1,000,000 candidates nationally.
WBJEE 2022 — Rank 318
Among 100,000 candidates in the state joint examination.
Atmadeep Young Scholars Award 2018
Awarded by The Times of India; ranked among the top 20 students in the state.
05

Skills & Coursework

Languages
PythonC++CSQLJavaScriptDartHTMLCSS
ML & Data
PyTorchNumPyPandasMatplotlibTransformersTensorFlow
Web & Backend
ReactNext.jsNode.jsFastAPIFlutterFirebasePostgreSQLMySQLMongoDB
Tools
GitGitHubDockerAWSGoogle ColabJupyterMATLABCOMSOLSolidWorks
Computer Science
Programming and Data StructuresAlgorithmsComputer Organization & ArchitectureComputer NetworksOOPDBMSOperating Systems
Machine Learning
AI Foundations and ApplicationsLinear AlgebraSoft Computing
Mathematics
Advanced CalculusPartial Differential EquationsTransform CalculusAutomata Theory
MOOCs
Neural Networks and Deep LearningComputer VisionPhysics-Informed Neural NetworksStructuring ML ProjectsBuilding LLMs
06

Positions of Responsibility

Secretary, Sports and Games · Hall of Residence, IIT Kharagpur

Nov 2023 — Jun 2024
  • Worked with the hall council to organize and manage sports events within the hostel.
  • Managed a ₹10,000 training budget, expanding the range of sports activities in the hall.