GeoShone: Towards A Geo-Foundation Model for Spatially Heterogeneous Network Modeling

Abstract

To achieve the ultimate goal of artificial general intelligence, an paradigm shift in artificial intelligence (AI) is burgeoning, moving from application-driven approach to generalist foundation models. Thus far, significant success of foundation models (e.g., GPT) has been observed in major AI applications, such as natural language processing. Motivated by this trend, there is an emerging interest in building the geospatial counterparts. By bridging to the established foundation models on text and image data, attempts have been made especially on remote sensing images. However, there is still a gap on non-euclidean geospatial data, dynamically generated on various spatial networks (e.g., transportation). In this study, harnessing the large-scale network dynamic data, we design a generic self-supervised pre-training technique (i.e., memory-augmented spatio-temporal contrastive learning) to develop the first geo-foundation model on spatial networks. In particular, the pre-trained model is empowered to handle the fundamental challenge of spatial heterogeneity and perform multiple node/graph-level downstream tasks with minimal fine-tuning. Extensive experiments on the real-world traffic benchmark datasets demonstrate the promising performance on geospatial tasks of different natures, including zero-shot classification of road types, few-shot spatial adaptation for traffic forecasting.