README.md 7.98 KB
Newer Older
José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
1
# S<sup>2</sup>-cGAN: Self-Supervised Adversarial Representation Learning for Binary Change Detection in Multispectral Images
ravanbakhsh's avatar
ravanbakhsh committed
2
3


ravanbakhsh's avatar
ravanbakhsh committed
4
This repository contains the code of the paper [S<sup>2</sup>-cGAN: Self-Supervised Adversarial Representation Learning for Binary Change Detection in Multispectral Images](https://arxiv.org/abs/2007.02565) accepted at the IEEE International Geoscience and Remote Sensing Symposium, 2020. This work has been done at the [Remote Sensing Image Analysis group](https://www.rsim.tu-berlin.de/menue/remote_sensing_image_analysis_group/) by [Jose Luis Holgado](), [Mahdyar Ravanbakhsh](https://www.rsim.tu-berlin.de/menue/team/dr_sayyed_mahdyar_ravanbakhsh/) and [Begüm Demir](https://begumdemir.com/).
matt's avatar
init  
matt committed
5
6
7
8

## Abstract
Deep Neural Networks have recently demonstrated promising performance in binary change detection (CD) problems in remote sensing (RS), requiring a large amount of labeled multitemporal training samples. Since collecting such data is time-consuming and costly, most of the existing methods rely on pre-trained networks on publicly available computer vision (CV) datasets. However, because of the differences in image characteristics in CV and RS, this approach limits the performance of the existing CD methods. To address this problem, we propose a self-supervised conditional Generative Adversarial Network (S<sup>2</sup>-cGAN). The proposed S<sup>2</sup>-cGAN is trained to generate only the distribution of unchanged samples. To this end, the proposed method consists of two main steps: 1) Generating a reconstructed version of the input image as an unchanged image 2) Learning the distribution of unchanged samples through an adversarial game. Unlike the existing GAN based methods (which only use the discriminator during the adversarial training to supervise the generator), the S<sup>2</sup>-cGAN directly exploits the discriminator likelihood to solve the binary CD task. Experimental results show the effectiveness of the proposed S<sup>2</sup>-cGAN when compared to the state of the art CD methods.

begumdemir's avatar
begumdemir committed
9
If you use our code, please cite the associated paper as:
José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
10

José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
11
> J. L. Holgado Alvarez, M. Ravanbakhsh, B. Demіr, "S<sup>2</sup>-cGAN: Self-Supervised Adversarial Representation Learning for Binary Change Detection in Multispectral Images", IEEE International Geoscience and Remote Sensing Symposium, Hawaii, USA, 2020.
José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
12
13
14
15
16

```
@article{IG203936,
  author={J.L. {Holgado Alvarez}, M. {Ravanbakhsh} and B. {Demіr}},
  journal={IGARSS 2020 - 2020 IEEE International Geoscience and Remote Sensing Symposium}, 
José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
17
  title={S^2-cGAN: Self-Supervised Adversarial Representation Learning for Binary Change Detection in Multispectral Images}, 
matt's avatar
init  
matt committed
18
  year={2020}
José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
19
20
}
```
ravanbakhsh's avatar
ravanbakhsh committed
21
## S<sup>2</sup>-cGAN Change Detection Strategy
José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
22
<!--![](test.png)-->
matt's avatar
init  
matt committed
23
24
25
26
27
28
29
30
31
 The following illustration represents the change detection strategy introduced in S<sup>2</sup>-cGAN.                                     
<img src="test.png"  width="450" height="200">



Examples of score maps associated to a pair of patches, and its comparison to the ground-truth.

<img src="results_table_igarss.png"  width="450" height="200">

ravanbakhsh's avatar
ravanbakhsh committed
32
For the details please check our [paper](https://arxiv.org/abs/2007.02565).
matt's avatar
init  
matt committed
33
34
35
36

## Setup
### Prerequisites
* The code is tested with Python 3.7, PyTorch 0.4.1 and Ubuntu 18.04. 
ravanbakhsh's avatar
ravanbakhsh committed
37
38
* Please, check out the [requirements](https://gitlab.tubit.tu-berlin.de/rsim/S2-cGAN/blob/master/requirements.txt) file for more information.
* For reproducibility, you can find our train and test sets [here](https://tubcloud.tu-berlin.de/s/AZYP4egBiWpxTK6) (password: Igarss_2020).
matt's avatar
init  
matt committed
39
40
41

### Hardware setup
The model was trained and tested in a machine with the following features:
José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
42
43
44
45
 * Nvidia Quadro P2000
 * 16 GB RAM
 * 12 CPU cores

matt's avatar
init  
matt committed
46
47
48
49
50
51
52
53
54
55
56
57
58
59
### Dataset
We used Worldview 2 satellite Very High Spatial Resolution (VHR) multispectral images dataset provided [here](https://github.com/MinZHANG-WHU/FDCNN).


### Construction of Train and Test Sets

```bash
 python data_creator.py 
```

The script `data_creator.py` expects the following arguments, you can used the default values or pass them as command arguments:
* `--band3` directory location of the rgb channels  
* `--band4` directory location of the I channel
* `--output_dir` output parent directory 
ravanbakhsh's avatar
ravanbakhsh committed
60
61
62
* `--test_dir` output directory for testset json file
* `--train_dir` output directory for trainset json file
* `--eval_dir` output directory for evaluation json file
José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
63
64
65
66
67
68
69
70
71
* `--gt` name and extension for ground-truth patches 
* `--A` name and extension for T1 patches 
* `--B`name and extension for T2 patches 
* `--city` identification code template for patches, by default wv2{}_{}. the first gap corresponds to tile id and the second gap correspond to patch id in the tile
* `--custom_filter_active` boolean variable to trigger filter operations over the trainset, the filter consist in remove patches from the trainset that contains a ratio of change/no_change bigger than certain threshold
* `--filter_value`, filter threshold value (0, 1)
* `--kernel` patch size, by default 128
* `--offset` padding value, by default 0
* `--stride` stride value, by default 10
matt's avatar
init  
matt committed
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111


### Training Phase

```bash
 python train.py --dataroot /patches --dataset /train_set/patch_coords.json --name S2_NET_001 
```

The script `train.py` expects the following command line arguments:
* `--dataroot` path to patches folder
* `--name` network name
* `--dataset` specify the path to the trainset json file


### Testing Phase

```bash
 python test.py --dataroot /patches --dataset /test_set/patch_coords.json --name S2_NET_001 --results_dir /results  
```

The script `test.py` expects the following command line arguments:
* `--dataroot` path to patches folder
* `--name` network name
* `--dataset` specify the path to the testset json file
* `--results_dir`  specify the path to the output directory



### Performance Evaluation
 
```bash
 python performance_evaluation.py  --dataroot /patches --dataset /perfomance_evaluation/perfomance_evaluation_coords.json --results_dir /results
```
 
The script `performance_evaluation.py` expects the following command line arguments:
* `--dataroot` path to patches folder
* `--dataset` specify the path to the perfomance_evaluation json file
* `--results_dir` specify the path to the output directory created by test.py


José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
112

José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
113
114
115
116

## Authors
**Jose Luis Holgado Alvarez**

José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
117
## Acknowledgements
matt's avatar
init  
matt committed
118
119
120
121
122
123
124
125
This software was developed based on:
P. Isola, J. Zhu, T. Zhou, A. A. Efros. [Image-to-Image Translation with Conditional Adversarial Nets]( https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix) , 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)


## Maintained by 
**Jose Luis Holgado Alvarez**
 
 E-mail:
José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
126
127
  * jose.l.holgadoalvarez@campus.tu-berlin.de
  * jlhalv92@gmail.com
José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
128
129

## License
matt's avatar
init  
matt committed
130
The code in this repository to facilitate the use of the `S2-cGAN: Self-Supervised Adversarial Representation Learning for Binary Change Detection in Multispectral Images` is licensed under the **MIT License**:
José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
131
132
133
134

```
MIT License

matt's avatar
init  
matt committed
135
Copyright (c) 2020 The Authors of The Paper, " S2-cGAN: Self-Supervised Adversarial Representation Learning for Binary Change Detection in Multispectral Images"
José Luis Holgado Álvarez's avatar
José Luis Holgado Álvarez committed
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
ravanbakhsh's avatar
ravanbakhsh committed
154
```