Skip to content
Snippets Groups Projects
Commit 2a540d63 authored by Arik Grahl's avatar Arik Grahl
Browse files

initial version

parents
Branches master
No related tags found
No related merge requests found
Expand all Hide whitespace changes
Inline Side-by-side
Dockerfile 0 → 100644
+ 71
− 0
View file @ 2a540d63
FROM alpine AS build
ARG ANDROID_TOOLS_VERSION=31.0.0
ARG ANDROID_TOOLS_REPO_BASE_URL=https://github.com/nmeum/android-tools
ENV ANDROID_TOOLS_URL=$ANDROID_TOOLS_REPO_BASE_URL/releases/download
ENV ANDROID_TOOLS_URL=$ANDROID_TOOLS_URL/$ANDROID_TOOLS_VERSION
ENV ANDROID_TOOLS_URL=$ANDROID_TOOLS_URL/android-tools
ENV ANDROID_TOOLS_URL=$ANDROID_TOOLS_URL-$ANDROID_TOOLS_VERSION.tar.xz
RUN \
apk update && \
apk upgrade && \
apk add \
brotli-dev \
build-base \
cmake \
go \
gtest-dev \
libusb-dev \
linux-headers \
lz4-dev \
pcre2-dev \
perl \
protobuf-dev \
zstd-dev \
&& \
rm -rfv /var/cache/apk/*
RUN \
wget $ANDROID_TOOLS_URL -O - | tar -xJ && \
cd android-tools-$ANDROID_TOOLS_VERSION && \
cmake \
-B build \
-DCMAKE_INSTALL_PREFIX=/usr \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_BUILD_TYPE=None && \
cmake \
--build build
FROM ruby:3.0.1-alpine
ARG ANDROID_TOOLS_VERSION=31.0.0
RUN \
apk update && \
apk upgrade && \
apk add \
brotli-libs \
libgcc \
libprotobuf \
libstdc++ \
libusb \
lz4-libs \
musl \
zlib \
zstd-libs \
rm -rfv /var/cache/apk/*
COPY --from=build \
/android-tools-$ANDROID_TOOLS_VERSION/build/vendor/adb \
/usr/bin/adb
WORKDIR /usr/src/app
COPY Gemfile Gemfile.lock /usr/src/app/
RUN bundle check || bundle install
COPY *.rb /usr/src/app/
ENTRYPOINT ["./measurement.rb"]
Gemfile 0 → 100644
+ 7
− 0
View file @ 2a540d63
source 'https://rubygems.org'
ruby '3.0.1'
gem 'chronic_duration'
gem 'ruby-progressbar'
gem 'slop'
Gemfile.lock 0 → 100644
+ 22
− 0
View file @ 2a540d63
GEM
remote: https://rubygems.org/
specs:
chronic_duration (0.10.6)
numerizer (~> 0.1.1)
numerizer (0.1.1)
ruby-progressbar (1.11.0)
slop (4.8.2)
PLATFORMS
x86_64-linux-musl
DEPENDENCIES
chronic_duration
ruby-progressbar
slop
RUBY VERSION
ruby 3.0.1p64
BUNDLED WITH
2.2.15
LICENSE 0 → 100644
+ 0
− 0
View file @ 2a540d63
This diff is collapsed.
README.md 0 → 100644
+ 100
− 0
View file @ 2a540d63
# Measurement
This repository contains the central `measurement` command line utility that can be used to perform automated network changes on mobile test devices and measure network performance metrics.
## Building
```shell
docker build \
-t measurement \
.
```
## Installation
For convenience, the wrapper script can be copied to the `$PATH` so that `measurement` is available as a command in the shell.
```shell
sudo cp ./measurement /usr/bin/measurement # or any location in the $PATH
```
## Usage
```
usage: measurement [options]
-h, --help print help message
-v, --version print version
```
### `run`
```
usage: measurement run [options]
-f, --file manifest file with measurement description
--dry-run only execute the measurement actions and do not send any data
-q, --quiet surpress output
-p, --protocol use specified protocols only
-h, --help print help message
```
### `analyze`
```
usage: measurement analyze [options]
-f, --file manifest file with measurement description
-o, --output output format, one of: csv, json, yaml
-O, --offset time offset
-p, --protocol use specified protocols only
-a, --moving-average time period of moving average
-d, --downtime analyze downtime durations from specified point in time
-h, --help print help message
```
## Measurement Manifests
The file [`example.yml`](https://git.tu-berlin.de/an-automated-test-system-for-quantifying-the-impact-of-mobile-network-changes-on-transmission-performance-metrics/measurement/-/blob/master/example.yml) contains an example manifest for a measurement.
```yaml
---
measurement:
name: 'demo measurement'
runs: 3
endpoint: '10.0.3.1'
iperf_path: '/data/local/tmp/iperf3.9'
actions:
edge:
- open: ['settings', 'wireless']
- tap: [540, 570] # mobile network
- tap: [540, 1850] # preferred network type
- swipe: [540, 340, 540, 2100] # swipe to top
- tap: [540, 490] # only GSM
- close: 'settings'
hspa+:
- open: ['settings', 'wireless']
- tap: [540, 570] # mobile network
- tap: [540, 1850] # preferred network type
- swipe: [540, 340, 540, 2100] # swipe to top
- tap: [540, 640] # only WCDMA
- close: 'settings'
lte:
- open: ['settings', 'wireless']
- tap: [540, 570] # mobile network
- tap: [540, 1850] # preferred network type
- swipe: [540, 340, 540, 2100] # swipe to top
- swipe: [540, 1610, 540, 1350] # swipe to LTE
- tap: [540, 1640] # only LTE
- close: 'settings'
steps:
- edge:
name: 'start measurement with EDGE'
duration: 10s
- hspa+:
name: 'upgrade to HSPA+'
duration: 10s
- lte:
name: 'finally switch to LTE'
duration: 10s
```
In addition to the metadata in the `measurement` key, each mobile standard is defined under `actions`.
The `steps` key defines the actual flow of the test, switching from EDGE to HSPA+ to LTE and staying in each step for 10 seconds.
analyze.rb 0 → 100644
+ 164
− 0
View file @ 2a540d63
measurements.each do |measurement|
name = measurement.dig('measurement', 'name')
name ||= Digest::SHA1.hexdigest(measurement.to_json)
runs = measurement.dig('measurement', 'runs')
runs ||= 1
interval = measurement.dig('measurement', 'interval')
interval ||= 0.1
interval = interval.to_f
sample_size = (@opts[:moving_average] / interval).round
sample_size = 1 if sample_size.zero?
total_duration = measurement['steps'].map do |step|
step.values.first['duration']
end
.sum
offset = @opts[:offset]
if @opts[:downtime]
data = {}
(1..runs).each do |run|
data[run] = {}
file = "#{DATA_DIR}#{name.parameterize}-#{run}-icmp.json"
next unless File.file?(file)
json = JSON.parse(File.read(file))
start_time = nil
times = [@opts[:downtime]]
json.each_with_index do |datum, index|
datum_time = DateTime.parse(datum['time']).to_time
start_time = datum_time if index.zero?
time = (datum_time - start_time + offset).round(1)
next if time < @opts[:downtime]
break if time > total_duration + offset
times << time
end
data[run]['icmp_downtime'] = times.each_cons(2).map { |a, b| b - a }.max if times.count > 1
times = [@opts[:downtime]]
file_glob = "#{DATA_DIR}#{name.parameterize}-#{run}-tcp*.json"
Dir.glob(file_glob).map do |filename|
JSON.parse(File.read(filename))
end
.sort_by do |json|
json.dig('start', 'timestamp', 'timesecs')
end
.each_with_index do |json, index|
file_time = DateTime.parse(json['start']['timestamp']['time'])
.to_time
start_time = file_time if index.zero?
datarates = []
json['intervals'].each_with_index do |json_interval, index|
time = (file_time - start_time).to_i + (index * interval) + offset
time = time.round(1)
next if time < @opts[:downtime]
break if time > total_duration + offset
json_interval['streams'].each do |stream|
next unless stream['sender']
datarates << stream['bits_per_second']
samples = datarates.last(sample_size)
datarate = samples.mean / ((sample_size + 1) - samples.size)
times << time if datarate > 0
end
end
end
data[run]['tcp_downtime'] = times.each_cons(2).map { |a, b| b - a }.max if times.count > 1
end
else
data = {}
from = (offset / interval).to_i
to = ((total_duration + offset) / interval).to_i
(from..to).each do |t|
data[(t * interval).round(1)] = {
'data_rate' => { 'tx' => [], 'rx' => [] },
'jitter' => [],
'packet_loss' => [],
'latency' => []
}
end
(1..runs).each do |run|
%w[tcp udp icmp].intersection(@opts[:protocol]).each do |protocol|
start_time = nil
if protocol == 'icmp'
file = "#{DATA_DIR}#{name.parameterize}-#{run}-#{protocol}.json"
next unless File.file?(file)
json = JSON.parse(File.read(file))
json_latencies = {}
json.each_with_index do |datum, index|
datum_time = DateTime.parse(datum['time']).to_time
start_time = datum_time if index.zero?
time = (datum_time - start_time + offset).round(1)
next unless data[time]
break if time > total_duration + offset
json_latencies[time] = datum['latency']
end
last_latency = nil
no_data_count = 0
data.each do |time, datum|
latency = json_latencies[time]
if latency
last_latency = latency
no_data_count = 0
else
no_data_count += 1
end
last_latency = nil if no_data_count >= 4
data[time]['latency'] << last_latency if last_latency
end
else
file_glob = "#{DATA_DIR}#{name.parameterize}-#{run}-#{protocol}*.json"
Dir.glob(file_glob).map do |filename|
JSON.parse(File.read(filename))
end
.sort_by do |json|
json.dig('start', 'timestamp', 'timesecs')
end
.each_with_index do |json, index|
file_time = DateTime.parse(json['start']['timestamp']['time'])
.to_time
start_time = file_time if index.zero?
datarates = { 'tx' => [], 'rx' => [] }
json['intervals'].each_with_index do |json_interval, index|
time = (file_time - start_time).to_i + (index * interval) + offset
time = time.round(1)
break if time > total_duration + offset
next unless data[time]
json_interval['streams'].each do |stream|
if protocol == 'tcp'
direction = stream['sender'] ? 'tx' : 'rx'
datarates[direction] << stream['bits_per_second']
samples = datarates[direction].last(sample_size)
datarate = samples.mean / ((sample_size + 1) - samples.size)
data[time]['data_rate'][direction] << datarate
elsif !stream['sender'] && protocol == 'udp'
if stream['jitter_ms']
data[time]['jitter'] << stream['jitter_ms']
end
if stream['lost_percent']
data[time]['packet_loss'] << stream['lost_percent']
end
end
end
end
end
end
end
end
data.each do |time, values|
values['data_rate']['tx'] = values['data_rate']['tx'].analyze(2, 0.0)
values['data_rate']['rx'] = values['data_rate']['rx'].analyze(2, 0.0)
values['jitter'] = values['jitter'].analyze(2, 100_000.0)
values['packet_loss'] = values['packet_loss'].analyze(1, 0.0)
values['latency'] = values['latency'].analyze(2, 100_000.0)
end
end
case @opts[:output]
when 'json'
puts data.to_json
when 'csv'
puts data.to_csv
else
puts data.to_yaml
end
end
example.yml 0 → 100644
+ 39
− 0
View file @ 2a540d63
---
measurement:
name: 'demo measurement'
runs: 3
endpoint: '10.0.3.1'
iperf_path: '/data/local/tmp/iperf3.9'
actions:
edge:
- open: ['settings', 'wireless']
- tap: [540, 570] # mobile network
- tap: [540, 1850] # preferred network type
- swipe: [540, 340, 540, 2100] # swipe to top
- tap: [540, 490] # only GSM
- close: 'settings'
hspa+:
- open: ['settings', 'wireless']
- tap: [540, 570] # mobile network
- tap: [540, 1850] # preferred network type
- swipe: [540, 340, 540, 2100] # swipe to top
- tap: [540, 640] # only WCDMA
- close: 'settings'
lte:
- open: ['settings', 'wireless']
- tap: [540, 570] # mobile network
- tap: [540, 1850] # preferred network type
- swipe: [540, 340, 540, 2100] # swipe to top
- swipe: [540, 1610, 540, 1350] # swipe to LTE
- tap: [540, 1640] # only LTE
- close: 'settings'
steps:
- edge:
name: 'start measurement with EDGE'
duration: 10s
- hspa+:
name: 'upgrade to HSPA+'
duration: 10s
- lte:
name: 'finally switch to LTE'
duration: 10s
measurement 0 → 100755
+ 11
− 0
View file @ 2a540d63
#!/usr/bin/env sh
docker run \
-it \
--rm \
--privileged \
-e DATA_DIR=/data \
-v `pwd`:/data \
-v /dev/bus/usb:/dev/bus/usb \
measurement \
$*
measurement.rb 0 → 100755
+ 82
− 0
View file @ 2a540d63
#!/usr/bin/env ruby
require 'chronic_duration'
require 'csv'
require 'date'
require 'digest/sha1'
require 'json'
require 'open3'
require 'ruby-progressbar'
require 'slop'
require 'timeout'
require 'yaml'
require './utils.rb'
VERSION = '0.1'.freeze
PROGRESSBAR_TIC = 10
PROGRESSBAR_FORMAT = '%t: %j%% [%b>%i] %e'.freeze
DATA_DIR = "#{ENV.fetch('DATA_DIR', Dir.getwd)}/"
command = ARGV.shift
@opts = case command
when 'run'
Slop.parse do |o|
o.array '-f', '--file',
'manifest file with measurement description',
required: true
o.bool '--dry-run',
'only execute the measurement actions ' \
'and do not send any data',
default: false
o.bool '-q', '--quiet',
'surpress output',
default: false
o.array '-p', '--protocol',
'use specified protocols only',
default: %w[tcp udp icmp]
o.on '-h', '--help', 'print help message' do
puts o
exit
end
end
when 'analyze'
Slop.parse do |o|
o.array '-f', '--file',
'manifest file with measurement description',
required: true
o.string '-o', '--output',
'output format, one of: csv, json, yaml',
default: 'yaml'
o.float '-O', '--offset',
'time offset',
default: 0.0
o.array '-p', '--protocol',
'use specified protocols only',
default: %w[tcp udp icmp]
o.float '-a', '--moving-average',
'time period of moving average',
default: 0.0
o.float '-d', '--downtime',
'analyze downtime durations from specified point in time',
default: nil
o.on '-h', '--help', 'print help message' do
puts o
exit
end
end
else
ARGV.unshift(command)
command = nil
Slop.parse do |o|
o.on '-h', '--help', 'print help message' do
puts o
exit
end
o.on '-v', '--version', 'print version' do
puts VERSION
end
end
end
require "./#{command}.rb" if command
run.rb 0 → 100644
+ 109
− 0
View file @ 2a540d63
check_for_android_devices
measurements.each do |measurement|
name = measurement.dig('measurement', 'name')
name ||= Digest::SHA1.hexdigest(measurement.to_json)
runs = measurement.dig('measurement', 'runs')
runs ||= 1
endpoint = measurement.dig('measurement', 'endpoint')
endpoint ||= 'iperf.eenet.ee'
iperf_path = measurement.dig('measurement', 'iperf_path')
iperf_path ||= '/usr/bin/iperf3'
interval = measurement.dig('measurement', 'interval')
interval ||= 0.1
interval = interval.to_f
total_duration = measurement['steps'].map do |step|
step.values.first['duration']
end
.sum
(1..runs).each do |run|
puts "executing run (#{run}/#{runs})" unless @opts[:quiet]
%w[tcp udp icmp].intersection(@opts[:protocol]).each do |protocol|
iperf_data = nil
ping_data = nil
start_time = Time.now
threads = []
threads << Thread.new do
if protocol == 'icmp'
ping_data = []
loop do
break if @opts[:dry_run]
duration = (total_duration - (Time.now - start_time)).ceil
break if duration <= 0
stdout, stderr, _ = adb_shell "ping -c 1 #{endpoint}"
/time=(?<latency>[0-9\.]+) ms/ =~ stdout
ping_data << {
time: DateTime.now.iso8601(3),
latency: latency.to_f,
log: stdout
} if latency
end
else
server_slot = 0
iperf_data = []
loop do
break if @opts[:dry_run]
duration = (total_duration - (Time.now - start_time)).ceil
break if duration <= 0
iperf_options = {
client: endpoint,
time: duration,
port: 5201 + (server_slot % 8),
interval: interval,
bidir: nil,
json: nil
}
iperf_options[:udp] = nil if protocol == 'udp'
stdout, stderr, _ = adb_shell iperf_path, iperf_options
results = parse_iperf_response(stdout)
iperf_data << results unless results['start']['connected'] == []
sleep 0.1 if stderr.match(/unable to/)
server_slot += 1 if stderr.match(/server is busy/)
end
end
end
measurement['steps'].each do |step|
action, parameters = step.to_a.flatten
threads << Thread.new do
perform_action measurement, action
end
unless @opts[:quiet]
title = "[#{protocol.upcase.to_s.ljust(4, ' ')}] " \
"#{parameters['name']}"
progressbar = ProgressBar.create(title: title,
total: parameters['duration'] * 10,
format: PROGRESSBAR_FORMAT)
end
(0...parameters['duration'] * 10).each do |tick|
sleep(0.1)
progressbar.increment unless @opts[:quiet]
end
end
threads.map(&:join)
if protocol == 'icmp'
filename = "#{DATA_DIR}" \
"#{name.parameterize}-#{run}-#{protocol}.json"
File.open(filename, 'w') do |file|
file.write(ping_data.to_json)
end
else
iperf_data.each_with_index do |datum, index|
filename = "#{DATA_DIR}" \
"#{name.parameterize}-#{run}-#{protocol}-#{index}.json"
File.open(filename, 'w') do |file|
file.write(datum.to_json)
end
end
end
end
end
end
utils.rb 0 → 100644
+ 222
− 0
View file @ 2a540d63
class String
def to_duration
ChronicDuration.parse(self.to_s)
end
def parameterize
self.to_s
.strip
.downcase
.gsub(/\s+/, '-')
.gsub('_', '-')
.gsub(/[^a-z0-9-]/, '')
.gsub(/-+/, '-')
.gsub(/-$/, '')
end
end
class Integer
def to_ordinal
return "#{self.to_s}\\\\textsuperscript{st}" if self == 1
return "#{self.to_s}\\\\textsuperscript{nd}" if self == 2
return "#{self.to_s}\\\\textsuperscript{rd}" if self == 3
"#{self.to_s}\\\\textsuperscript{th}"
end
end
class Array
def to_csv
csv = CSV.generate do |csv|
self.to_a.each do |row|
csv << row
end
end
csv.to_s
end
end
class Hash
def to_csv
csv = CSV.generate do |csv|
headers = self.headers([], [], self.values.inject(&:merge))
csv << [''] + headers.map { |items| items.join('_') }
self.each do |x, x_values|
row = [x]
headers.each do |header|
row << x_values.dig(*header)
end
csv << row
end
end
csv.to_s
end
def headers(all_keys, keys, object)
if object.is_a?(Hash)
object.each do |key, value|
all_keys = self.headers(all_keys, keys + [key], value)
end
else
all_keys << keys
end
all_keys
end
end
module Enumerable
def sum
self.inject(0) { |accum, i| accum + i }
end
def mean
return nil if self.empty?
self.sum / self.length.to_f
end
def sample_variance
m = self.mean
sum = self.inject(0) { |accum, i| accum +(i - m)**2 }
return 0.0 if self.length == 1
sum / (self.length - 1).to_f
end
def standard_deviation
Math.sqrt(self.sample_variance)
end
def median
self.percentile(0.5)
end
def percentile(percentage)
another_array = self.to_a.dup
another_array.push(-1.0 / 0.0)
another_array.sort!
another_array_size = another_array.size - 1
r = percentage.to_f * (another_array_size - 1) + 1
if r <= 1
return another_array[1]
elsif r >= another_array_size
return another_array[another_array_size]
end
ir = r.truncate
fr = r - r.truncate
another_array[ir] + fr * (another_array[ir + 1] - another_array[ir])
end
def analyze(min_samples = 1, no_data = 0.0)
percentiles = [0.01, 0.1, 0.25, 0.5, 0.75, 0.9, 0.99]
{
'percentiles' => percentiles.each_with_object({}) do |percentage, hash|
hash[percentage] = self.to_a.count >= min_samples ? self.to_a.percentile(percentage) : no_data
end,
'mean' => self.to_a.mean || no_data,
'standard_deviation' => self.to_a.standard_deviation || no_data,
'sample_variance' => self.to_a.sample_variance || no_data,
'sample_size' => self.to_a.size || 0
}
end
end
def parse_iperf_response(stream)
warning = 'WARNING: Size of data read does not correspond to offered length'
JSON.parse(stream.gsub(warning, ''))
end
def adb_init
`adb get-state 2> /dev/null`.strip == 'device'
end
def adb_shell(command, options = {})
arguments = options.map { |key, value| "--#{key} #{value}" }.join(' ')
Open3.capture3("adb shell #{command} #{arguments}")
end
def perform_action(measurement, action)
measurement['actions'][action]&.each do |action|
action = action.to_a.first
command = action.shift
arguments = action.flatten
case command
when 'open'
case arguments
when ['settings']
adb_shell 'am start -a android.settings.SETTINGS'
when ['settings', 'wireless']
adb_shell 'am start -a android.settings.WIRELESS_SETTINGS'
end
sleep 0.2
when 'close'
case arguments
when ['settings']
adb_shell 'am force-stop com.android.settings'
end
sleep 0.2
when 'tap'
adb_shell "input tap #{arguments.first(2).join(' ')}"
sleep 0.2
when 'swipe'
adb_shell "input swipe #{arguments.first(4).join(' ')}"
sleep 0.2
when 'press'
case arguments
when ['home']
adb_shell 'input keyevent 3'
end
sleep 0.2
end
end
end
def exponential_backoff(try, max = 32)
backoff = 2**try
backoff = max if backoff > max
backoff
end
def check_for_android_devices
puts 'checking for online devices, ' \
'please confirm debugging prompt on device' unless @opts[:quiet]
(0..5).each do |try|
return if adb_init
backoff = exponential_backoff(try)
puts "retrying in #{backoff} seconds ..."
sleep backoff
end
puts 'did not find device, check USB connectivity, enable USB debugging ' \
'and confirm debugging prompt on device' unless @opts[:quiet]
exit 1
end
def measurements
@measurements ||= if @opts[:file] == ['-']
$stdin.read
.split('---')
.map { |yaml| YAML.load(yaml) }
else
@opts[:file].map do |file|
YAML.load_file("#{DATA_DIR}#{file}")
end
end
.select { |yaml| yaml.is_a?(Hash) }
.map do |measurement|
measurement['steps']&.each do |step|
duration = step.values.first['duration'].to_duration
step.values.first['duration'] = duration
end
measurement
end
end
def check_measurement_status(measurement_data)
return unless measurement_data
json = JSON.parse(measurement_data)
if json['start']['connected'] == []
puts 'cannot connect to iperf server, ' \
'please check connectivity' unless @opts[:quiet]
exit 1
end
end
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment