From Social Media to Visual Insights: A Multimodal Data Mining Approach for Hurricane

Keywords: social media, multimodal, data mining, geoAI, hurricane, natural disaster, cartographic visualization
Abstract Type: Paper Abstract

Abstract

Recent advancements in spatial analytics and geospatial artificial intelligence (GeoAI) have opened new possibilities for using social media data, especially from platforms like Twitter (now X), in natural disaster management. This research explores how user-generated text and images provide valuable spatiotemporal insights to aid disaster response and strengthen community resilience. Despite the potential of social media data in emergencies, a gap remains in studies that integrate GeoAI and cartographic visualization for quickly mining and utilizing this data to improve situational awareness.

Recent machine learning technologies, particularly those processing multimodal data, hold promise for enhancing disaster management applications. The Contrastive Language–Image Pre-training (CLIP) model by OpenAI, for example, integrates text and image analysis to give responders a more comprehensive understanding of crises for adaptive interventions.

This study evaluates the effectiveness of models like CLIP in processing Twitter’s text and image data across multiple hurricane incidents. Findings confirm that multimodal approaches excel in tasks such as informativeness assessment, humanitarian category classification, and damage evaluation. Applied to tweets from Hurricane Harvey in 2017, this method produced both static and dynamic cartographic visualizations, offering a detailed view of disaster dynamics.

In summary, this research shows that the CLIP-based multimodal approach effectively integrates social media texts and images, advancing disaster management by enabling vital information extraction and visualization for future innovations.