CrossEarth: Geospatial Vision Foundation Model for Domain Generalizable Remote Sensing Semantic Segmentation

Ziyang Gong^{1 ∗}, Zhixiang Wei^{2 ∗}, Di Wang^{3 ∗}, Xianzheng Ma³, Hongruixuan Chen⁴, Yuru Jia⁵⁶, Yupeng Deng¹, Zhenming Ji^{1 †}, Xiangwei Zhu^{1 †}, Naoto Yokoya⁴, Jing Zhang³, Bo Du³, Liangpei Zhang³

¹ Sun Yat-sen University, ² University of Science and Technology of China, ³ Wuhan University,

⁴ The University of Tokyo, ⁵ KU Leuven, ⁶ KTH Royal Institute of Technology

^∗ Equal contribution, ^† Corresponding author

• [2024/11/06] The most checkpoints have been uploaded and you can access them in the huggingface badges.

• The environment and inference steps please refer to the following installation. The inference codes and weights will be coming soon.

• The benchmark collection in the paper is releasing and you can access it at here.

• 🎉🎉🎉 CrossEarth is the first VFM for Remote Sensing Domain Generalization (RSDG) semantic segmentation. We just release the arxiv paper of CrossEarth. You can access CrossEarth at here.

• Visualization
• Environment Requirements
• Inference steps
• Training steps
• Model Weights with Configs

In Radar figure:

• CrossEarth achieves SOTA performances on 23 evaluation benchmarks across various segmentation scenes, demonstrating strong generalizability.

In UMAP figures:

• CrossEarth extracts features that cluster closely for the same class across different domains, forming well-defined groups in feature space, demonstrating its ability to learn robust, domain-invariant features.

• Moreover, CrossEarth features exhibit high inter-class separability, forming unique clusters for each class and underscoring its strong representational ability to distingguish different categories.

conda create -n CrossEarth -y
conda activate CrossEarth
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia -y
pip install -U openmim
mim install mmengine
mim install "mmcv>=2.0.0"
pip install "mmsegmentation>=1.0.0"
pip install "mmdet>=3.0.0"
pip install xformers=='0.0.20' 
pip install -r requirements.txt
pip install future tensorboard

First, download the model weights from the huggingface or Baidu Netdisk in the above badges. Notably, the checkpoints of dinov2_converted.pth and dinov2_converted_1024x1024.pth are needed for inference. Please download them and put them in the CrossEarth/checkpoints folder.

Second, change the file path in experiment config files (configs/base/datasets/xxx.py and configs/CrossEarth_dinov2/xxx.py), and run the following command to inference. (Take 512x512 inference as an example)

python tools/test.py configs/CrossEarth_dinov2/CrossEarth_dinov2_mask2former_512x512_bs1x4.py ./checkpoints/xxx.pth

Notably, save path of pseudo labels is in the experiment config file. When testing CrossEarth on different benchmarks, you also need to change the class number in CrossEarth_dinov2_mask2former.py file.

Coming soon.

If you find CrossEarth helpful, please consider giving this repo a ⭐ and citing:

@article{crossearth,
  title={CrossEarth: Geospatial Vision Foundation Model for Domain Generalizable Remote Sensing Semantic Segmentation},
  author={Gong, Ziyang and Wei, Zhixiang and Wang, Di and Ma, Xianzheng and Chen, Hongruixuan and Jia, Yuru and Deng, Yupeng and Ji, Zhenming and Zhu, Xiangwei and Yokoya, Naoto and Zhang, Jing and Du, Bo and Zhang, Liangpei},
  journal={arXiv preprint arXiv:2410.22629},
  year={2024}
}

• MTP: Advancing remote sensing foundation model via multi-task pretraining
• Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation