Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
1 result
Name Last commit Last update
backbones
compression
datasets
hashing
mtl
sample_run
utilities
README.md
bench_MLRSNet.py
eval.py
figure.png
parameters.py
sci-cbir.png
train_C.py
train_SCI.py
README.md

Towards Simultaneous Image Compression and Indexing for Scalable Content-Based Retrieval in Remote Sensing

This repository contains code of the paper Towards Simultaneous Image Compression and Indexing for Scalable Content-Based Retrieval in Remote Sensing. This work has been done at the Remote Sensing Image Analysis group by Gencer Sumbul, Jun Xiang and Begum Demir .

If you use this code, please cite our paper given below:

G. Sumbul, J. Xiang, B.Demir, "Towards Simultaneous Image Compression and Indexing for Scalable Content-Based Retrieval in Remote Sensing", IEEE Transactions on Geoscience and Remote Sensing, 2022.

@article{JointCompHashing,
  author={G. {Sumbul}, J. {Xiang}, and B. {Demіr}},
  journal={IEEE Transactions on Geoscience and Remote Sensing}, 
  title={Towards Simultaneous Image Compression and Indexing for Scalable Content-Based Retrieval in Remote Sensing}, 
  year={2022},
  doi={10.1109/TGRS.2022.3204914}
}

This repository contains (in parts) code that has been adapted from:

Introduction

Due to the rapidly growing remote sensing (RS) image archives, RS images are usually stored in a compressed format for reducing their storage sizes. Thus, most of the existing content-based RS image retrieval systems require fully decoding of images that is computationally demanding for large-scale RS image archives. To address this issue, we introduce a novel approach devoted to simultaneous RS image compression and indexing for scalable content-based image retrieval (denoted as SCI-CBIR). The proposed SCI-CBIR prevents the requirement of decoding RS images prior to image search and retrieval. To this end, it includes two main steps: i) deep learning-based compression; and ii) deep hashing-based indexing. The first step effectively compresses RS images by employing a pair of deep encoder and decoder neural network and an entropy model. The second step produces hash codes with a high discrimination capability for RS images by employing pairwise, bit-balancing and classification loss functions. For the training of the proposed SCI-CBIR approach, we also introduce a novel multi-stage learning procedure with automatic loss weighting techniques to characterize RS image representations, which are appropriate for both RS image indexing and compression. The proposed learning procedure enables automatically weighting different loss functions considered for the proposed approach instead of computationally demanding grid search.

Prerequisites

The code in this repository requires Python 3.7.6 and pytorch 1.7.0.

In addition, Range coder is required for lossless compression of the latents. The code is tested in Ubuntu 20.04.

The following is an exemplary setup which contains everything needed:

(1) wget  https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
(2) bash Miniconda3-latest-Linux-x86_64.sh (say yes to append path to bashrc)
(3) source .bashrc
(4) sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update && sudo apt-get install g++
(5) sudo apt-get install libgdal-dev gdal-bin
(6) ogrinfo --version
(7) conda activate base
(9) conda install matplotlib scipy scikit-learn scikit-image tqdm pillow pytorch  
(7) pip install wandb glymur pybind11 openpyxl faiss-gpu
(10) pip install --global-option=build_ext --global-option="-I/usr/include/gdal/" GDAL==<GDAL VERSION FROM OGRINFO>
(11) python ./hashing-and-compression/compression/cpp_exts/setup.py  build
(12) python ./hashing-and-compression/compression/cpp_exts/setup.py  install

Datasets

Downloaded data should be placed in a folder named Dataset and keep the original structure as following:

Dataset
└───BigEarthNet
|    └───S2A_MSIL2A_20170613T101031_0_48
|           │   S2A_MSIL2A_20170613T101031_0_48_B0
|           │   ...
|    ...
|
└───MLRSNet
|    |   Categories_names.xlsx
|    └───Images
|    |      └───airplane
|    |              │   airplane_00001.jpg
|    |              │   ...
|    └───labels
|    |      └───airplane.csv
|    ...

Note: The train/val/test splits of each dataset and its subsets are placed in ./hashing-and-compression/datasets.

During training or evaluation, memory-map files are created from the dataset for fast processing.

To use the subset data, set the flag --flag_subset.

Logging results with W&B

  • Create an account here (free): https://wandb.ai
  • After the account is set, make sure to include your API key in parameters.py under --wandb_key.
  • Set the flag --log_online to use wandb logging, if the network is unavailable in the training environment, set the flag --wandb_dryrun to make wandb store the data locally, and upload the data with the command wandb sync <$path/wandb/offline..>
  • Set --project, --group and --savename during the training.

Common setups

The following are common setups for all the training scripts:

  • --source_path: Path for the dataset, eg. $path/Dataset.
  • --dataset_name': Dataset name, choose between BigEarthNet and MLRSNet.
  • --save_path: Path to save everything.
  • --flag_subset: Flag. If set, select subset dataset.
  • --flag_checkpoint: Flag. If set, apply torch.utils.checkpoint to save GPU memory.
  • --epochs: Total number of epochs for training.

Training and optimization of our proposed SCI-CBIR approach

The training and optimization of our proposed SCI-CBIR approach done by a multi-stage learning procedure:

  • Learning reconstruction and bit-rate optimization: DL-Based Compression is optimized for getting a good reconstruction first followed by the bit-rate control using MGDA. These two stages are implemented in train_C.py
  • Learning hash codes: The Deep Hashing-Based Indexing is added to the compression and trained together. The hashing loss is automatically updated using PCGrad or DWA (by setting the flag) and the compression is updated by a small learning rate. This final stage of SCI-CBIR is implemented in train_SCI.py.

All the parameters for training are listed and explained in parameters.py.

A set of exemplary runs is provided in ./sample_run.

If the training is stopped accidentally, it can be resumed by setting --load_from_checkpoint, load the check point from $save_path/savename/every_epoch.pth.tar.

Detailed parameter settings for our multi-stage SCI-CBIR optimization

Stage 1 & 2: Train the DL based compression for a wide range of bit-rates

The script train_C.py expects the following command line arguments:

  • --noBpp_epoch: Epochs which will not backprop Bpp loss. It sets the number of epochs for Learning reconstruction stage. Here the compression is trained for the distortion loss only to get a good quality reconstruction. After this it adds the bit-rate optimization.

Note:
When the current epoch is smaller than --noBpp_epoch, the checkpoint is saved for the best PSNR on the validation set.

When the current epoch is larger than --noBpp_epoch, the checkpoint is saved for current smallest bit-rate on the validation set.

 # Command to train DL-based compression with stage 1 and 2 optimization
 python  train_C.py --source_path 'path_to_dataset' --dataset_name BigEarthNet --save_path 'path_to_save_results' --flag_subset  --epochs 1500 --noBpp_epoch 120 --log_online --project  'project_name' --group  'grp_name'  --savename  'save_name'

Stage 3: Adds the deep hashing based indexing along with the compression and optimize simultaneously

The script train_SCI.py expects the following command line arguments:

  • --load_from_checkpoint: Path of the pretrained compression model from stage 2.
  • --flag_start_new: Flag. If set, reset the initial epoch to 0 and log everything in a new folder.
  • --flag_PCGrad: Flag. If set, the hashing sublosses are optimized by PCGrad.
  • --num_samples_per_class: Int. If set to 0, the batch data is randomly sampled. Otherwise, the batch data is sampled by the class balance samper. Default to 2.
  • --flag_augumentation: Flag. If set, apply random horizontal or vertical flip to input images

Note: Stage 3 loads the pretrained compression part, and train the comrpession and hashing simultaneously.

The checkpoint is saved for the best averaged precision on the validation set.

 # Command to train SCI-CBIR with Stage 3 optimization. Here the compression is optimized by MGDA and the hashing is optimized by PCGrad/DWA depending on the flag.
 python train_SCI.py --source_path 'path_to_dataset' --dataset_name BigEarthNet --save_path 'path_to_save_results'  --flag_subset  --flag_augumentation   --epochs 40 --load_from_checkpoint 'path_to_pretrained_compression_stage2'  --flag_start_new --flag_PCGrad --log_online --project 'project_name' --group 'grp_name' --savename 'save_name '

Evaluation

The evaluation results are written into the checkpoint folder.

The script eval.py expects the following command line arguments:

  • --dataset_path: Path for the dataset, eg. $path/Dataset/BigEarthNet.
  • --metrics: List of metrics for evaluating retrieval performance.
  • --load_from_checkpoint: Path of the pretrained models (compression or hashing or both).
  • --entropy_estimation: Flag. If set, use estimated entropy to evaluate the compression performance.
  • --cuda: Flag. If set, use cuda during evaluation.
# Evaluate the compression and retrieval performance of SCI-CBIR approach
 python eval.py --dataset_path 'path_to_dataset_with_datasetname'  --flag_subset --cuda --load_from_checkpoint 'pretrained_simultaneous _model_from_stage3'

Authors

Jun Xiang xj.junxiang@gmail.com

License

The code in this repository to facilitate the use of the Towards Simultaneous Image Compression and Indexing for Scalable Content-Based Retrieval in Remote Sensing is licensed under the MIT License:

MIT License

Copyright (c) 2022 The Authors of The Paper, "Towards Simultaneous Image Compression and Indexing for Scalable Content-Based Retrieval in Remote Sensing"

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.