README.md 6.95 KB
Newer Older
ahmetkerem's avatar
ahmetkerem committed
1
# A Consensual Collaborative Learning Method for Remote Sensing Image Classification Under Noisy Multi-Labels
Ahmet Kerem Aksoy's avatar
Ahmet Kerem Aksoy committed
2

ahmetkerem's avatar
ahmetkerem committed
3
This repository contains code of the paper `A Consensual Collaborative Learning Method for Remote Sensing Image Classification Under Noisy Multi-Labels`. This work has been developed developed at the [Remote Sensing Image Analysis group](https://www.rsim.tu-berlin.de/menue/remote_sensing_image_analysis_group/) by Ahmet Kerem Aksoy, Mahdyar Ravanbakhsh, Tristan Kreuziger and Begüm Demir.
ahmetkerem's avatar
ahmetkerem committed
4
5

If you use this code, please cite our paper given below:
ahmetkerem's avatar
ahmetkerem committed
6
> Ahmet Kerem Aksoy, Mahdyar Ravanbakhsh, Tristan Kreuziger, Begüm Demir, "[A Consensual Collaborative Learning Method for Remote Sensing Image Classification Under Noisy Multi-Labels](https://arxiv.org/abs/2105.05496)", IEEE International Conference on Image Processing, Alaska, USA, Sep. 2021.
ahmetkerem's avatar
ahmetkerem committed
7

ravanbakhsh's avatar
ravanbakhsh committed
8
9
10

![](ICIP_2021.mp4)

akakream's avatar
akakream committed
11
12
![](images/CCML_Code_Figure.png)

tristan.kreuziger's avatar
tristan.kreuziger committed
13
_CCML intuition: For a batch of input images, the models `f` and `g` need to agree on a subset, which contains only the correctly annotated images._
akakream's avatar
akakream committed
14

tristan.kreuziger's avatar
tristan.kreuziger committed
15
## Description
akakream's avatar
akakream committed
16

ravanbakhsh's avatar
ravanbakhsh committed
17
We propose a multi-label learning method based on the idea of co-training for scene classification of remote sensing (RS) images with noisy labels. Our proposed Consensual Collaborative Multi-Label Learning (CCML) method identifies, ranks and corrects training images with noisy multi-labels through four main modules: 1) discrepancy module; 2) group lasso module; 3) flipping module; and 4) swap module. The discrepancy module ensures that the two networks learn diverse features, while obtaining the same predictions. The group lasso module detects the potentially noisy labels by estimating the label uncertainty based on the aggregation of two collaborative networks. The flipping module corrects the identified noisy labels, whereas the swap module exchanges the ranking information between the two networks.
akakream's avatar
akakream committed
18

Ahmet Kerem Aksoy's avatar
Ahmet Kerem Aksoy committed
19
## Dependencies
tristan.kreuziger's avatar
tristan.kreuziger committed
20
The code in this repository has been tested with `Python 3.7.6`. To run it, the following packages must be installed:
tristan.kreuziger's avatar
tristan.kreuziger committed
21
22
23
24
25
26
- `tensorflow==2.3.0`
- `tensorflow-gpu==2.3.0`
- `tensorflow-addons==0.11.2`
- `noisifier==0.4.4`
- `scipy==1.5.4`
- `matplotlib==3.3.3`
akakream's avatar
akakream committed
27
28
- `PyYAML==5.3.1`
- `h5py==2.10.0`
Ahmet Kerem Aksoy's avatar
Ahmet Kerem Aksoy committed
29

30
## Datasets
ahmetkerem's avatar
ahmetkerem committed
31
We have tested the method on two different RS datasets: [BigEarthNet](http://bigearth.net/) and [UC Merced Land Use dataset](http://weegee.vision.ucmerced.edu/datasets/landuse.html). This repository supports different versions of these datasets:
tristan.kreuziger's avatar
tristan.kreuziger committed
32
33
34
- BigEarthNet with a 12 class-nomenclature
- BigEarthNet with a 19 class-nomenclature
- BigEarthNet with a 43 class-nomenclature
ahmetkerem's avatar
ahmetkerem committed
35
- UC Merced Land Use dataset annotated with multi-labels provided [here](https://bigearth.eu/datasets)
Ahmet Kerem Aksoy's avatar
Ahmet Kerem Aksoy committed
36

tristan.kreuziger's avatar
tristan.kreuziger committed
37
To use BigEarthNet, three [TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) are needed: `train.tfrecord`, `val.tfrecord` and `test.tfrecord`. The path to the directory containing the three files must be provided as `dataset_path` in the command line arguments.
Ahmet Kerem Aksoy's avatar
Ahmet Kerem Aksoy committed
38

ahmetkerem's avatar
ahmetkerem committed
39
The UC Merced Land Use dataset is loaded through [pickle](https://docs.python.org/3/library/pickle.html) files. To use it, 6 pickle files must be created: `namely x_train.pickle`, `x_validation.pickle`, `x_test.pickle`, `y_train.pickle`, `y_validation.pickle`, `y_test.pickle`. The directory with these 6 files must be provided as `dataset_path` in the command line arguments.
akakream's avatar
akakream committed
40

akakream's avatar
akakream committed
41
## Synthetic Label Noise
42
For testing purposes, we introduced synthetic label noise to the datasets using the python package [noisifier](https://git.tu-berlin.de/rsim/noisifier). The `Random Noise per Sample` label noise approach from the package is applied to the dataset labels.  
akakream's avatar
akakream committed
43

akakream's avatar
akakream committed
44
45
## Arguments

tristan.kreuziger's avatar
tristan.kreuziger committed
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
- `batch_size`: The size of the mini-batches that are sampled in each epoch from the dataset.
- `epoch`: The number of epochs for the training.
- `arch` : Possible architectures are `resnet`, `denseNet`, `SCNN`, `paper_model`, `keras_model`, `modified_SCNN`, `batched_SCNN`. 
- `dataset_path`: The path to the data as desribed above.
- `channel` : Possible values: `RGB`, `ALL`. Use only `RGB` for UC Merced. Use `ALL` to use 10 bands of BigEarthNet. 
- `label` : Possible values are `BEN-12`, `BEN-19`, `BEN-43`, `ucmerced`. 
- `sigma` : Sigma for the rbf kernel in mmd. 
- `swap` : If 1, swap between models, if 0 do not swap. 
- `swap_rate` : The percentage of swap between models. If there is no swap, set to 1.
- `lambda2` : Strength of the dicrepancy component, a value between 0 and 1.
- `lambda3` : Strength of teh consistency component, a value between 0 and 1.
- `flip_bound` : The percentage of unflipped training epochs.
- `flip_per` : Class flipping percentage determing how much of the consented classes is flipped.
- `miss_alpha` : Rate of missing label noise to be included into the error loss. More missing labels are detected, when this value is high.
- `extra_beta` : Rate of extra label assignment noise to be included into the error loss. More extra label assignments are detected, when this value is high.
- `add_noise` : Add noise to the dataset. Enter 0 for no noise; enter 1 for adding noise.
- `noise_type` : Choose the noise type to be added to the dataset. Possible values are 1 for Random Noise
63
per Sample and 2 for Mix Label Noise.
tristan.kreuziger's avatar
tristan.kreuziger committed
64
65
66
67
68
- `sample_rate` : Percentage of samples in a mini-batch to be noisified, a value between 0 and 1.
- `class_rate` : Percentage of labels in a sample to be noisified, a value between 0 and 1. Mix Label Noise does not use this.
- `metric` : Possible metrics: `mmd`, `shannon`, `wasserstein`, `nothing`.
- `alpha` : Influence of the group lasso ranking loss compared to binary cross entropy for detection of noisy samples.
- `test` : 1 to test the model using only a small portion of the datasets. Default is 0. 
akakream's avatar
akakream committed
69

tristan.kreuziger's avatar
tristan.kreuziger committed
70
An example configuration can be found in the `run.sh` file under the `scripts` directory.
Ahmet Kerem Aksoy's avatar
Ahmet Kerem Aksoy committed
71
72

## How to run
akakream's avatar
akakream committed
73
`pip install -r requirements.txt && mkdir output && cd script/ && ./run.sh`
74

ahmetkerem's avatar
ahmetkerem committed
75
76
## Authors
Ahmet Kerem Aksoy
Ahmet Kerem Aksoy's avatar
Ahmet Kerem Aksoy committed
77

ahmetkerem's avatar
ahmetkerem committed
78
Tristan Kreuziger
Ahmet Kerem Aksoy's avatar
Ahmet Kerem Aksoy committed
79

80
81
82
83
84
85
## License
The code in this repository is licensed under the **MIT License**:

```
MIT License

ravanbakhsh's avatar
ravanbakhsh committed
86
Copyright (c) 2021 Ahmet Kerem Aksoy
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```