PhD Candidate · KAIST · Daejeon, South Korea

Sumit
Mishra

Multimodal Foundation Models & Vision-Language Learning for Safety-Critical Systems

Deep learning researcher at KAIST's Vehicular Intelligence Lab, specializing in CLIP-based systems, cross-modal knowledge distillation, and explainable AI. Building perception systems that bridge foundational multimodal research with real-world road safety deployment.

Google Scholar LinkedIn ↓ CV ↓ Resume
Sumit Mishra
13+
Journal Papers
3
Patents
Profile

About Me

I am a PhD candidate in Robotics at KAIST (Korea Advanced Institute of Science and Technology), affiliated with the Vehicular Intelligence Lab under Prof. Dongsoo Har. My thesis — "Explainable AI for Safer Roads Using Multimodal Framework" — investigates how vision-language foundation models can make autonomous perception systems both accurate and interpretable.

My work spans self-supervised visual-language learning, cross-modal knowledge distillation, concept bottleneck models, and edge-AI optimization. I have published in IEEE Transactions on ITS (IF 9.1), ISPRS (IF 12.9), and other top venues, with two papers featured in KAIST Breakthrough.

Before KAIST, I spent several years as a research consultant on smart-city AI projects in India, collaborating with Deloitte India and securing grants from the University of Edinburgh and the Government of India. My Indian Patent (No. 538566) is deployed in Hyderabad and Bangalore for adaptive traffic signaling.

Research Interests
  • Vision-Language & Multimodal Foundation Models (CLIP, BLIP, LLaVA)
  • Zero-shot, Few-shot & Weakly Supervised Learning
  • Cross-modal Knowledge Transfer & Alignment
  • Explainable AI in Computer Vision
  • Safety-Critical Autonomous Perception Systems
  • Edge-AI Optimization & Model Compression
  • Intelligent Transportation Systems (ADAS, VAD)
Education
2023 – 2026 (Expected)
Ph.D. in Robotics
KAIST  ·  GPA 3.85/4.3 (95.5%)
Thesis: Explainable AI for Safer Roads Using Multimodal Framework
2021 – 2023
M.S. in Robotics
KAIST  ·  GPA 4.0/4.3 (97%)
Thesis: Multi-modal Data Usage for Smart City Mobility Services
2012 – 2016
B.Tech. Electronics & Communication
Galgotias College of Engineering, AKTU
Scholarship

Research Publications

IEEE Transactions on ITS · 2026
Accident-Prone Road Feature Screening with Spatial and Textual Interpretability
Mishra, S., Mishra, M., Ko, K.W., and Har, D.
IF: 9.1
IEEE Transactions on ETCI · Under Review
Zero-Shot Road Anomaly Detection Based on Text Corpus Knowledge and Segmentation-First Localization
Mishra, S., Kumar, H., Shyam, P., Har, D., and Ko, K.W.
Under Review
ISPRS Journal of Photogrammetry and Remote Sensing · Elsevier 2026
Cross-Modal Distillation for Real-time Wildfire Detection and Localization in Edge-Deployed Aerial Vehicles
Mishra, M., Mishra, S., and Shin, H.
IF: 12.9
IEEE Signal Processing Letters · 2026
IMUDistill: Knowledge Transfer for Enhancing Low Precision IMU Performance
Mishra, S., Kumar, H., Ko, K.W., and Har, D.
IF: 3.9
IEEE ITS Magazine · 2025
A Secure, Blockchain-Enabled Vehicular Sensor Communication Protocol With Deep Learning-Assisted Anomaly Detection
Mishra, S., Sengar, N., and Har, D.
IF: 5.5
IEEE Transactions on ITS · 2025
TIME-VAD: Text-Informed Magnitude Enhancement Feature Learning for Vehicle Accident Detection
Mishra, S., Mishra, M., Shyam, P., and Har, D.
KAIST BreakthroughIF: 9.1
IEEE Sensors Journal · 2024
Integrating Multi-sourced Sensor Data for Enhanced Traffic State Estimation
Mishra, M., Mishra, S., and Har, D.
IF: 4.3
IEEE Transactions on ITS · 2023
Sensing Accident-Prone Features in Urban Scenes for Proactive Driving and Accident Prevention
Mishra, S., Rajendran, P.K., Santos V, L.F., and Har, D.
KAIST BreakthroughIF: 9.1
Transportation Letters · Taylor & Francis 2022
Adaptive Traffic Signal Control in Developing Countries Using Parameters from Crowd-Source Data
Mishra, S., Singh, V., Gupta, A., Bhattacharya, D., and Mudgal, A.
IF: 3.6
ISPRS Int. Journal of Geo-Information · MDPI 2021
Application Based COVID-19 Micro-Mobility Solution for Safe and Smart Navigation in Pandemics
Mishra, S., Singh, N., and Bhattacharya, D.
IF: 3.4
Data · MDPI 2018
Congestion Adaptive Traffic Light Control and Notification Architecture Using Google Maps APIs
Mishra, S., Bhattacharya, D., and Gupta, A.
IF: 2.4
Journal of Cleaner Production · Elsevier 2019
Comparative Emission Study by Real-time Congestion Monitoring for Stable Pollution Policy
Mishra, S., Kushwaha, A., Aggrawal, D., Gupta, A.
IF: 11.1
Optik · Elsevier 2016
Optical Parameters Testing to Redefine Visibility for Low Cost Transmissometer Using Channel Modeling
Mishra, S. and Ahuja, B.
IF: 3.1
ECCV 2022
RELMOBNET: A Robust Two-Stage End-To-End Training Approach for MobileNetV3 Based Relative Camera Pose Estimation
Rajendran, P.K., Mishra, S., Santos V, L.F., and Har, D.
IEEE Intelligent Vehicles Symposium (IV) 2022
Infra Sim-to-Real: An Efficient Baseline and Dataset for Infrastructure-Based Online Object Detection and Tracking Using Domain Adaptation
Shyam, P., Mishra, S., Yoon, K.J., and Kim, K.S.
ACM SIGSPATIAL IWCTS 2022
Assessing Micro-Mobility Services in Pandemics for Studying 10-Minutes Cities Concept in India
Mishra, S., Bhattacharya, D., Chaturvedi, A., and Singh, N.
RiTA 2021
Socially Acceptable Route Planning and Trajectory Behavior Analysis of Personal Mobility Device
Mishra, S., Rajendran, P.K., and Har, D.
arXiv · 2302.07440
RoadSafe: A Road Redesign Technique Achieving Enhanced Road Safety by Diffusion Model Inpainting
Mishra, S., Mishra, M., Kim, T., and Har, D.
ISPRS Annals · SDSC-2018
Adaptive Traffic Light Cycle Time Controller Using Microcontrollers and Crowdsource Data of Google APIs
Mishra, S., Bhattacharya, D., Gupta, A., Singh, V.R.
IEEE INDICON 2015
Network Security Protocol For Constrained Resource Devices In Internet Of Things
Mishra, S.
IP & Deployment

Patents & Deployed Systems

Indian Patent No. 538566 · Granted May 2024
System for Spatio-Temporal Traffic Data Reporting for Smart City Services
Gupta, A. and Mishra, S.  ·  View Patent Document  ·  Deployed in Hyderabad & Bangalore  ·  Covered by Times of India & Amar Ujala
Korean Patent Application 10-2024-0103095 · Pending
Real-Time Traffic Monitoring and Analysis System
Mishra, S., Mishra, M., Har, D.  ·  KAIST-supported filing
Korean Patent Application 10-2024-0087193 · Pending
Accident Probability Prediction System
Mishra, S., Mishra, M., Har, D.  ·  KAIST-supported filing
Key Research

Featured Projects

IEEE T-ITS · Under Review
VisText-Net: Explainable Accident-Prone Feature Discovery
Dual-branch CLIP-based concept bottleneck model combining CNN and CLIP text encoder. Automated concept discovery via mechanistic interpretability with counterfactual validation and optimal transport (Sinkhorn) for patch-to-concept assignment.
64.25% confidence drop39–63% CAM ablationCLIP + CBM
IEEE T-ITS 2025 · IF 9.1
TIME-VAD: Text-Informed Vehicle Accident Detection
Reversed CLIP magnitude assumption for accident detection. Magnitude-Enhanced MIL with text-anchor contrastive loss and Dilated Temporal Conv-Attention (DTCA) block. Strong cross-domain generalization (CCD to DAD).
94.44% ROC-AUC (DoTA)+9.74% over SOTA99.97% AP (CCD)
IEEE T-ETCI · Under Review
Zero-Shot Road Anomaly Detection
Triple-CLS architecture with controlled noise injection for cross-modal synchronization. Zero-shot captioning without paired training. Systematic benchmarking of CLIP, BLIP, LLaVA for edge deployment trade-offs.
92.42% accuracy+16–22% over VLMs15–25x smaller model
ISPRS 2026 · IF 12.2
Cross-Modal Wildfire Detection (Edge AI)
Thermal-to-optical knowledge transfer: frozen CLIP teacher to trainable CNN student. Spatial distillation via Pearson correlation, semantic distillation via cosine affinity. Optimized for aerial edge deployment.
100% image-level accuracy2.99 GFLOPS0.004s inference
Career

Work Experience

Jun 2025 – Nov 2025
Remote Research Intern
Amazon Fulfillment Technologies & Robotics · Supervised by Dr. Pranjay Shyam (Applied Scientist II)
  • Built zero-shot anomaly detection (92.42% accuracy) with Triple-CLS architecture for cross-modal alignment
  • Evaluated foundation models (CLIP, BLIP, LLaVA) for edge deployment — accuracy-efficiency trade-offs
  • Contributed TIME-VAD framework achieving 94.44% on DoTA, 9.74% over prior SOTA
Mar 2021 – Present
Graduate Research Assistant
Vehicular Intelligence Lab, KAIST · Advisor: Prof. Dongsoo Har
  • Leading 7 papers on multimodal learning (2 under review: IEEE T-ITS, IEEE T-ETCI; 5 published, IF: 4.3–12.2)
  • Developed 3 CLIP-based frameworks: VisText-Net (CBMs), TIME-VAD, wildfire detection (cross-modal KD)
  • Achieved SOTA across 6 datasets with edge-deployment feasibility (2–3 GFLOPS)
  • Deployed accident-prone feature notification POC with Deloitte India (Lucknow region)
  • Led COVID-19 safe routing portal with Deloitte, GIZ, Univ. of Edinburgh — secured £11,500 grant
Jun 2018 – Feb 2021
Research Consultant
Learnogether Technologies Pvt. Ltd.
  • Led smart-city AI projects on traffic management and urban mobility
  • Collaborated with Deloitte India on adaptive traffic signaling in Gurugram
  • Secured £11,500 (University of Edinburgh) and Rs.12.5 lakh (MSME India) in research grants
  • Developed drowsy driver detection and eye-tracking for commercial transportation safety
Aug 2016 – Jun 2018
Research Scholar
Galgotias College of Engineering and Technology
  • Developed real-time congestion mapping for Delhi region (similar to Google Environmental Insights)
  • POC for traffic light control, IoT network security, and low-cost visibility measurement
  • Published 4 papers; co-invented Indian Patent No. 538566 (deployed in Hyderabad & Bangalore)
Technical

Core Expertise

Vision-Language Foundation Models
CLIPDINOFlorenceBLIPLLaVAConcept Bottleneck ModelsZero-shot LearningFew-shot LearningPrompt EngineeringCross-modal Alignment
Multimodal Learning
Cross-modal Knowledge DistillationContrastive LearningSimCLRMoCoText-anchor Contrastive LossWeak SupervisionMulti-instance Learning
Deep Learning & Computer Vision
PyTorchTensorFlowVision Transformers (ViT, DeiT)Attention MechanismsOpenCVDetectron2MMDetectionGrad-CAM / CAMDomain AdaptationEdge-AI OptimizationModel Compression
Tools & Infrastructure
Python (8+ years)Hugging FaceGitDockerWeights & BiasesTensorBoardJupyter
Recognition

Awards & Grants

Community

Academic Service

Reviewer — Conferences
IEEE Intelligent Vehicles Symposium (IV) ACM CHI — Conference on Human Factors in Computing Systems
Reviewer — Journals
IEEE Transactions on Intelligent Transportation Systems IEEE Sensors Journal IEEE Access Array Mobile Networks and Applications Multimedia Tools and Applications Scientific Reports Qeios Cluster Computing Peer-to-Peer Networking and Applications
Invited Talks
Multi-Modal Data Usage for Smart City Use Cases Especially Traffic Surveillance
GCET
AI/ML Applications in Road Safety and Traffic
IIT BHU
Get in Touch

Contact & References

KAIST Email
Personal Email
Phone
(+82) 10-4493-5168
Location
KAIST, Daejeon 34141, South Korea
Google Scholar LinkedIn ↓ Download CV ↓ Download Resume
References
Prof. Dongsoo Har
PhD Advisor · CCS Graduate School of Mobility, KAIST
Dr. Pranjay Shyam
Applied Scientist II · Amazon Fulfillment Technologies & Robotics, Seattle
Dr. Luiz Felipe Vecchietti
Senior Researcher · Max Planck Institute for Security & Privacy, Germany
Prof. Ankit Gupta
MoRTH Chair Professor · Dept. of Civil Engineering, IIT (BHU)
Dr. Devanjan Bhattacharya
MSCA TRAIN@ED Fellow · University of Edinburgh