Learning Across Decentralized Multi-Modal Remote Sensing Archives with Federated Learning
This repository contains code of the paper abstract Learning Across Decentralized Multi-Modal Remote Sensing Archives with Federated Learning submitted to IEEE International Geoscience and Remote Sensing Symposium (IGARSS) 2023. This work has been done at the Remote Sensing Image Analysis group by Baris Buyuktas, Gencer Sumbul, and Begüm Demir.
This repository contains (in parts) code that has been adapted from:
Introduction
Remote sensing (RS) image archives are stored under different databases due to their growth in size and the data storage limitations of gathering all the data in a centralized server. In addition, RS image archives of some data providers (e.g., commercial providers) may not be accessible to the public due to commercial concerns, legal regulations, etc. However, most of the deep learning (DL) based approaches require full access to data while learning the model parameters of deep neural networks (DNNs). When there is no access to data on decentralized RS image archives, federated learning (FL) can be used, which aims to learn DNN models on distributed databases (i.e., clients) and to find the optimal model parameters in a global server (i.e., global model) without accessing data on clients. However, RS images on different clients can be associated with different data modalities. To address the above-mentioned issue, as the first time in RS, we propose a FL framework in the context of multi-modal scene classification in RS. The proposed framework aims to learn DNNs on a central server for multi-modal classification problems without accessing data on clients, which can hold RS images associated with multiple modalities. To this end, our framework includes three modules: 1) multi-modal fusion (MF); 2) feature whitening (FW); and 3) mutual information maximization (MIM). According to the results, our framework achieves the highest score compared to MSFedAvg, which is a way to perform multi-modal FL by separately performing FedAvg algorithm for multiple modalities and then averaging class probabilities during inference.
Prerequisites
The code in this repository requires Python 3.10.4, pytorch 1.12.1.
The code is tested in Ubuntu 18.04.
An exemplary setup which contains everything needed:
(1) wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
(2) bash Miniconda3-latest-Linux-x86_64.sh (say yes to append path to bashrc)
(3) source .bashrc
(4) sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update && sudo apt-get install g++
(5) sudo apt-get install libgdal-dev gdal-bin
(6) ogrinfo --version
(7) conda activate base
(8) conda install matplotlib scipy scikit-learn scikit-image tqdm pillow pytorch
Training Arguments
-
epoch: number of local training -
round: number of communication rounds -
clients: number of total clients -
lr: learning rate -
batch: local batch size -
decay: weight decay -
optimizer: type of optimizer -
sampling: sampling rate -
step_size: step size -
gamma: learning rate decay per global round
How to run
python main.py
Authors
Baris Buyuktas https://rsim.berlin/team/members/baris-buyuktas
Gencer Sümbül https://rsim.berlin/team/members/gencer-sumbul
License
The code in this repository to facilitate the use of the Learning Across Decentralized Multi-Modal Remote Sensing Archives with Federated Learning is licensed under the MIT License:
MIT License
Copyright (c) 2023 The Authors of The Paper, "Learning Across Decentralized Multi-Modal Remote Sensing Archives with Federated Learning"
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.