Commit e09f7049 authored by Safaa Alnabulsi's avatar Safaa Alnabulsi
Browse files

Add first draft of code docs

parent 6d8b47c6
<!doctype html>
<html class="no-js">
<head>
<meta charset="utf-8">
<title>Seminar: Real-time Image Smoothing via Iterative Least Squares</title>
<meta name="description" content="">
<meta name="viewport" content="width=device-width">
<!-- Place favicon.ico and apple-touch-icon.png in the root directory -->
<link href="http://fonts.googleapis.com/css?family=Raleway:300,400,600" rel="stylesheet" type="text/css">
<link rel="stylesheet" type="text/css" href="style.css">
<script type="text/javascript" async
src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-MML-AM_CHTML"></script>
<!--<link rel="stylesheet" href="styles/main.37ab405b.css">-->
<body>
<!--[if lt IE 7]>
<p class="browsehappy">You are using an
<strong>outdated</strong> browser. Please
<a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.
</p>
<![endif]-->
<div class="container">
<nav class="navbar">
<div class="container">
<ul class="navbar-list">
<li class="navbar-item">
<a class="navbar-link" href="#intro">Intro</a>
</li>
<li class="navbar-item">
<a class="navbar-link" href="#problem">Other Smoothing Methods</a>
</li>
<li class="navbar-item">
<a class="navbar-link" href="#ILS">This Paper</a>
</li>
<li class="navbar-item">
<a class="navbar-link" href="#code">Code</a>
</li>
<li class="navbar-item">
<a class="navbar-link" href="#math">Math</a>
</li>
<li class="navbar-item">
<a class="navbar-link" href="#references">References</a>
</li>
</ul>
</div>
</nav>
<section class="header" id="intro">
<h2 class="title">Seminar: Real-time Image Smoothing via Iterative Least Squares</h2>
<h6>Seminar by Safaa Alnabulsi (<a href="mailto:safaa.alnabulsi@campus.tu-berlin.de">safaa.alnabulsi@campus.tu-berlin.de</a>)
</h6>
<div class="row">
<!-- <div class="one column category" style="text-align: center;">-->
<!-- <h5 class="docs-header">Teaser Image</h5>-->
<!-- <img class="u-max-full-width" src="images/teaser.jpg">-->
<p>
Smoothing is a technique in Image processing which is used to reduce noise in Images or to produce a
less pixelated image.
It's an essetinal key which enhances images used in several domains and applications. There are several
methods and algorithms in the literature which performs smoothing on images e.g. weighted average based
methods, pyramid based methods, deep learning based methods ..etc
There is a tradeoff between the smoothing quality and the processing speed: the high smoothing quality
usually requires a high computational cost which leads to the low processing speed.
In this paper, we will see a new global optimization based method, named iterative least squares (ILS),
for efficient edge-preserving image smoothing with much lower computational cost.
</p>
</div>
</section>
<div class="docs-section" id="problem">
<h3 class="section-heading">Weighted Average Based Smoothing Methods</h3>
<p class="section-description">
Before digging into this paper's enhancements, let's take a look on the results of other smoothing methods
which are used in Opencv
You will see from the following results the poor quality of smoothing methods performed by weighted average
based methods.
</p>
<div class="row">
<div class="column category" style="text-align: center;">
<h5 class="docs-header">Results of weighted average based methods</h5>
<img class="u-max-full-width" src="images/all-methods.png">
<p>
</p>
</div>
</div>
<h5> How much noise is reduced? </h5>
<p class="section-description">
After taking a look into the reduced noise results performed by each of the filters above, one will notice
that most of the important features of the image are subtracted.
That is not optimal as one aims to keep the important features of an image while subtracting only
unimportant details i.e. noise.
This lead us to a conclusion that those methods, regardless of their small execuation time, are not an
optimal way to get the results one aspire to.
</p>
<div class="row">
<div class="one-third column category" style="text-align: center;">
<h5 class="docs-header">Averaging - 2D Convolution</h5>
<img class="u-max-full-width" src="images/averaging.png">
<p>An empty image.</p>
</div>
<div class="one-third column category" style="text-align: center;">
<h5 class="docs-header">Averaging - Blur</h5>
<img class="u-max-full-width" src="images/blur.png">
<p>An second empty image.</p>
</div>
<div class="one-third column category" style="text-align: center;">
<h5 class="docs-header">Gaussian Filtering</h5>
<img class="u-max-full-width" src="images/gblur.png">
<p>A super empty image.</p>
</div>
<div class="one-third column category" style="text-align: center;">
<h5 class="docs-header">Median Filtering</h5>
<img class="u-max-full-width" src="images/median.png">
<p>An empty image.</p>
</div>
<div class="one-third column category" style="text-align: center;">
<h5 class="docs-header">Bilateral Filtering</h5>
<img class="u-max-full-width" src="images/bilateral.png">
<p>An second empty image.</p>
</div>
</div>
<h5> Execution Time </h5>
<div class="row">
<div class="column category" style="text-align: center;">
<h5 class="docs-header">Execution Time</h5>
<table class="u-full-width">
<thead>
<tr>
<th>Method</th>
<th>Execution Time</th>
</tr>
</thead>
<tbody>
<tr>
<td>Averaging - 2D Convolution</td>
<td>0.00913</td>
</tr>
<tr>
<td>Averaging - Blur</td>
<td>0.00433</td>
</tr>
<tr>
<td>Gaussian Filtering</td>
<td>0.01631</td>
</tr>
<tr>
<td>Median Filtering</td>
<td>0.00642</td>
</tr>
<tr>
<td>Bilateral Filtering</td>
<td>0.05833</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<div class="docs-section" id="ILS">
<h5>Real-time Image Smoothing via Iterative Least Squares</h5>
<div class="u-pull-right" style="text-align: center; width:300px; margin-left:20px">
<img class="u-max-full-width" src="images/original-paper.png">
<p>Original Image</p>
<img class="u-max-full-width" src="images/ILS-paper.png">
<p>After applying ILS</p>
</div>
<p>
<!-- This text can float around <a href="#figure2">figure 2</a>.-->
This paper propses a new edge-preserving smoothing method which smoothes out small details in images and
preserves the major edges and structures.
Among these approaches, two kinds of methods have been widely developed: weighted average based methods and
global optimization based methods. </p>
<p>
<div class="row">
<div class="one-half column category">
<h5 class="docs-header">Benefits of ILS</h5>
<ul>
<li>High-quality results.</li>
<li>It can produce results with little visible artifacts.</li>
<li>Lower computational cost.</li>
<li>The computation of ILS is simple and highly parallel</li>
<li>The ILS is flexible; it can be modified to handle more applications that require different
smoothing properties
</li>
</ul>
</div>
</div>
</p>
<div class="row">
<div class="column category" style="text-align: center;">
<h5 class="docs-header">ILS Single CPU</h5>
<img class="u-max-full-width" src="images/ils-cpu-single.png">
</div>
<div class="column category" style="text-align: center;">
<h5 class="docs-header">ILS Parallel CPU with Numpy</h5>
<img class="u-max-full-width" src="images/ils-numpy.png">
</div>
<div class="column category" style="text-align: center;">
<h5 class="docs-header">ILS Parallel CPU with Scipy</h5>
<img class="u-max-full-width" src="images/ils-scipy.png">
</div>
</div>
<h5> Execution Time </h5>
<p>It is clear from the results below that the time of parallel execution is ### less than one single CPU.
Regarding the parallel cpu, I also compared the performance of two libraries: numpy and scipy.
Scipy outperformed Numpy with ###.
</p>
<div class="row">
<div class="column category" style="text-align: center;">
<table class="u-full-width">
<thead>
<tr>
<th>Method</th>
<th>Execution Time</th>
</tr>
</thead>
<tbody>
<tr>
<td>Single CPU</td>
<td>2.1222</td>
</tr>
<tr>
<td>Parallel CPU with Numpy</td>
<td>1.4564</td>
</tr>
<tr>
<td>Parallel CPU with Scipy</td>
<td>1.3981</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<div class="docs-section" id="code">
<h3 class="section-heading">Code</h3>
<p class="section-description">At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd
gubergren, no sea takimata ita sanctus est Lorem ipsum dolor sit amet.</p>
<pre>
<code>
def _fft(img, interface='single'):
if interface == 'single':
out = np.fft.fft2(img,axes=(0,1))
elif interface == 'parallel_numpy':
out = pyfftw.interfaces.numpy_fft.fft2(img,axes=(0,1))
elif interface == 'parallel_scipy':
out = pyfftw.interfaces.scipy_fftpack.fft2(img,axes=(0,1))
return out
def _ifft(img, interface='single'):
if interface == 'single':
out = np.fft.ifft2(img,axes=(0,1)).real
elif interface == 'parallel_numpy':
out = pyfftw.interfaces.numpy_fft.ifft2(img,axes=(0,1)).real
elif interface == 'parallel_scipy':
out = pyfftw.interfaces.scipy_fftpack.ifft2(img,axes=(0,1)).real
return out
</code>
<code>
def psf2otf_Dx(outSize, interface):
psf = np.zeros(outSize)
psf = psf.astype('float32')
psf[0, 0] = -1
psf[0, -1] = 1
otf = _fft(psf, interface)
return otf
def psf2otf_Dy(outSize, interface):
psf = np.zeros(outSize)
psf = psf.astype('float32')
psf[0, 0] = -1
psf[-1, 0] = 1
otf = _fft(psf, interface)
return otf
</code>
<code>
def ILS_Norm(F,c,lam, interface='single'):
N, M, D = F.shape # (768, 1024, 3)
sizeI2D = [N, M]
# --------------------------------------------------------------------------------------------
# pre-compute
otfFx = psf2otf_Dx(sizeI2D,interface)
otfFy = psf2otf_Dy(sizeI2D,interface)
Denormin = np.abs(otfFx)**2 + np.abs(otfFy)**2 # take the real part and to power of two
Denormin = Denormin[:,:, np.newaxis] # add a third axis
Denormin = np.repeat(Denormin, D, axis=2) # repeat the same data D=3 times on the third axis=2
denominator = 1 + 0.5 * c * lam * Denormin
# --------------------------------------------------------------------------------------------
U = F
normin1 = _fft(U, interface)
# --------------------------------------------------------------------------------------------
for i in range(4):
## Step 1: eq 7 - Intermediate variables \mu update, in x-axis and y-axis direction
# Gradients delta u on x axis and delta u on y axis
# image gradient in x direction
u_extra_col = (U[:,0,:] - U[:,-1,:])[:,np.newaxis,:]
u_extra_row = (U[0,:,:] - U[-1,:,:])[np.newaxis,:,:]
u_h = np.hstack((np.diff(U,1,1), u_extra_col))
u_v = np.vstack((np.diff(U,1,0), u_extra_row))
mu_h = c * u_h - p * u_h * (u_h * u_h + eps)**gamma
mu_v = c * u_v - p * u_v * (u_v * u_v + eps)**gamma
### ---------------------------------------------------------------------------- ##
## Step 2: eq 9 - Update the smoothed image U
# The diff causes loss in one columns
extra_col = (mu_h[:,-1,:] - mu_h[:, 0,:])[:,np.newaxis,:]
extra_row = (mu_v[-1,:,:] - mu_v[0,:,:])[np.newaxis,:,:]
# we calculate the diff - the inverse first order derivative of μxn along x-axis & μyn along y-axis
normin2_h = np.hstack((extra_col , - np.diff(mu_h,1,1)))
normin2_v = np.vstack((extra_row , - np.diff(mu_v,1,0)))
fft_normin_h_v = _fft(normin2_h + normin2_v, interface)
numerator = normin1 + 0.5 * lam * fft_normin_h_v
FU = numerator / denominator
U = _ifft(FU, interface)
# U = np.abs(ifft2(FU,axes=(0,1)))
normin1 = FU
return U
</code>
<code>
# Constants
lam = 1
k = 5
p = 0.8
eps = 0.0001
gamma = 0.5 * p - 1
c = p * eps**gamma
F = read_image('flower.png')
</code>
<code>
start = timer()
smoothed1 = ILS_Norm(F,c,lam,interface='single')
print("ILS_Norm CPU Single:", timer()-start)
show_image(F, smoothed1,t='Smoothed Image Single CPU')
</code>
<code>
start = timer()
smoothed2 = ILS_Norm(F,c,lam, interface='parallel_numpy')
print("ILS_Norm CPU Parallel Numpy:", timer()-start)
show_image(F, smoothed2,t='Smoothed Image Parallel Numpy')
</code>
<code>
start = timer()
smoothed3 = ILS_Norm(F,c,lam,interface='parallel_scipy')
print("ILS_Norm CPU Parallel Scipy:", timer()-start)
show_image(F, smoothed3,t='Smoothed Image Parallel Scipy')
</code>
</pre>
<p>
Inline code snippets <code>are also possible</code> as this beautiful <code>super code i++; 0x34</code>
shows.
</p>
</div>
<div class="docs-section" id="math">
<h3 class="section-heading">Math</h3>
<h4 class="docs-header">Objective Function</h4>
<img class="u-max-full-width" src="images/objective-function.png">
<h4 class="docs-header">ILS problem</h4>
<img class="u-max-full-width" src="images/ILS-problem.png">
</div>
<div class="docs-section" id="references">
<h3 class="section-heading">References</h3>
<ul class="popover-list">
<li class="popover-item" id="1">
<!--A link to this may look like: <a href="#1">[1]</a>-->
[1] Wei Liu and Pingping Zhang and Xiaolin Huang and Jie Yang and Chunhua Shen and Ian Reid, Real-time
Image Smoothing via Iterative Least Squares, <i> presented at SIGGRAPH 2020,
2020</i>.
</li>
<li class="popover-item" id="2">
[2] <a
href="https://opencv24-python-tutorials.readthedocs.io/en/stable/py_tutorials/py_imgproc/py_filtering/py_filtering.html">Smoothing
Images in OpenCV</a>
</li>
</ul>
</div>
</div>
/*! Template based on the following projects:
* normalize.css v3.0.2 | MIT License | git.io/normalize
*
* Skeleton V2.0.4
* Copyright 2014, Dave Gamache
* www.getskeleton.com
* Free to use under the MIT license.
* http://www.opensource.org/licenses/mit-license.php
*/
html {
font-family: sans-serif;
-ms-text-size-adjust: 100%;
-webkit-text-size-adjust: 100%
}
body {
margin: 0
}
article,
aside,
details,
figcaption,
figure,
footer,
header,
hgroup,
main,
menu,
nav,
section,
summary {
display: block
}
audio,
canvas,
progress,
video {
display: inline-block;
vertical-align: baseline
}
audio:not([controls]) {
display: none;
height: 0
}
[hidden],
template {
display: none
}
a {
background-color: transparent
}
a:active,
a:hover {
outline: 0
}
abbr[title] {
border-bottom: 1px dotted
}
b,
strong {
font-weight: 700
}
dfn {
font-style: italic
}
h1 {
font-size: 2em;
margin: .67em 0
}
mark {
background: #ff0;
color: #000
}
small {
font-size: 80%
}
sub,
sup {
font-size: 75%;
line-height: 0;
position: relative;
vertical-align: baseline
}
sup {
top: -.5em
}
sub {
bottom: -.25em
}
img {
border: 0
}
svg:not(:root) {
overflow: hidden
}
figure {
margin: 1em 40px
}
hr {
box-sizing: content-box;
height: 0
}
pre {
overflow: auto
}
code,
kbd,
pre,
samp {
font-family: monospace, monospace;
font-size: 1em
}
button,
input,
optgroup,
select,
textarea {
color: inherit;
font: inherit;
margin: 0
}
button {
overflow: visible
}
button,
select {
text-transform: none
}
button,
html input[type=button],
input[type=reset],
input[type=submit] {
-webkit-appearance: button;
cursor: pointer
}
button[disabled],
html input[disabled] {
cursor: default
}
button::-moz-focus-inner,
input::-moz-focus-inner {
border: 0;
padding: 0
}
input {
line-height: normal
}
input[type=checkbox],
input[type=radio] {
box-sizing: border-box;
padding: 0
}
input[type=number]::-webkit-inner-spin-button,
input[type=number]::-webkit-outer-spin-button {
height: auto
}
input[type=search] {
-webkit-appearance: textfield;
box-sizing: content-box
}
input[type=search]::-webkit-search-cancel-button,
input[type=search]::-webkit-search-decoration {
-webkit-appearance: none
}
fieldset {
border: 1px solid silver;
margin: 0 2px;
padding: .35em .625em .75em
}
legend {
border: 0;
padding: 0