From Social Media to Visual Insights: A Multimodal Data Mining Approach for Hurricanes

Keywords: multimodal data mining, geoAI, social media, hurricane, natural disaster, cartographic visualization
Abstract Type: Paper Abstract

Abstract

Advancements in spatial analytics and geospatial artificial intelligence (GeoAI) have opened new possibilities for leveraging social media data, such as public platforms like Twitter (now X), for natural disaster management. Despite the potential of social media data in emergencies, a gap remains in studies integrating GeoAI and cartographic visualization to efficiently mine and utilize this dataset for improving situational awareness. This research explores how user-generated texts and images provide valuable spatiotemporal insights to aid disaster response and enhance community resilience during hurricane events.

Machine learning technologies, particularly those processing multimodal data, hold promise for enhancing disaster management applications. One notable advancement is the Contrastive Language–Image Pre-training (CLIP) model by OpenAI. The CLIP model can interpret text and images within a joint embedding space, providing instant insights that empower emergency responders and authorities to better understand crises and make more informed, adaptive decisions.

In this study, multiple CLIP models are applied to a pre-labeled Twitter dataset specifically designed for disaster-related research, encompassing three specific tasks relevant to disaster response. Using Hurricane Harvey as a case study, this research first compares different CLIP models based on Twitter texts, images, and their integration. After statistical validation, specific model will be selected for each task and further implemented to extract disaster-related information from a distinct Twitter dataset of the same event. Following model application and necessary data cleaning, various charts and cartographic visualizations are generated to address disaster-related questions, contributing to a more comprehensive understanding of the complex dynamics of this event.