Takeru Ohta sile

starzを使って集計:

前回: 2019/04/10

$ starz sile

Total: 2087

jsone                      ★  250

Benchmark Result Report

Report ID: ecb7024f149c2653a906f600d048eac2658f94bc2cfc8e50bfd7356c0fb1e29b
Kurobako Version: 0.1.3
Number of Solvers: 4
Number of Problems: 2
Metrics Precedence: best value -> AUC

Please refer to ["A Strategy for Ranking Optimizers using Multiple Criteria"][Dewancker, Ian, et al., 2016] for the ranking strategy used in this report.

Benchmark Result Report

Report ID: 4ba0c3ff34e08ddaac04387c13628701e21fd683a713b496b3f78c3d541beac9
Kurobako Version: 0.1.3
Number of Solvers: 3
Number of Problems: 4
Metrics Precedence: best value -> AUC

Please refer to ["A Strategy for Ranking Optimizers using Multiple Criteria"][Dewancker, Ian, et al., 2016] for the ranking strategy used in this report.

The aim of this benchmark is to compare the performances of Optuna's pruners (i.e., NopPruner, MedianPruner, SuccessiveHalvingPruner and the ongoing HyperbandPruner). All of the pruners were used by the default settings in this benchmark.

The commands to execute this benchmark.

// (1) Downloads `kurobako` (BBO benchmark tool) binary.
$ curl -L https://github.com/sile/kurobako/releases/download/0.1.3/kurobako-0.1.3.linux-amd64 -o kurobako
$ chmod +x kurobako && sudo mv kurobako /usr/local/bin/

// (2) Downloads data files of HPOBench. (notice that the total size is over 700MB)
$ curl -OL http://ml4aad.org/wp-content/uploads/2019/01/fcnet_tabular_benchmarks.tar.gz

$ cargo install kurobako
// or (only linux)
$ wget https://github.com/sile/kurobako/releases/download/0.0.15/kurobako-0.0.15.linux-amd64 -o kurobako && chmod +x kurobako
// or 
$ git clone git://github.com/sile/kurobako.git && cd kurobako && git checkout 0.0.14 && cargo install --path .

// 独自サンプラの場合
$ kurobako benchmark --problems (kurobako problem-suite sigopt auc) --solvers (kurobako solver command -- python3 /tmp/optuna_solver_example.py ) --budget 100 --iterations 10 | kurobako run > /tmp/sigopt-my-sampler.json

	# A solver implementation based on Random Search algorithm.
	from kurobako import problem
	from kurobako import solver
	import numpy as np

	class RandomSolverFactory(solver.SolverFactory):
	def specification(self):
	return solver.SolverSpec(name='Random Search')

	def create_solver(self, seed, problem):

	# A solver implementation based on Simulated Annealing algorithm.
	from kurobako import solver
	from kurobako.solver.optuna import OptunaSolverFactory
	import optuna

	class SimulatedAnnealingSampler(optuna.BaseSampler):
	# Please refer to
	# https://github.com/optuna/optuna/blob/v1.0.0/examples/samplers/simulated_annealing_sampler.py
	# for the implementation.
	...

	# 1. Download kurobako binary.
	$ curl -L https://github.com/sile/kurobako/releases/download/0.2.6/kurobako-0.2.6.linux-amd64 -o kurobako
	$ chmod +x kurobako && sudo mv kurobako /usr/local/bin/

	# 2. Download the data file for HPOBench (note that the file size is about 700MB).
	$ curl -OL http://ml4aad.org/wp-content/uploads/2019/01/fcnet_tabular_benchmarks.tar.gz
	$ tar xf fcnet_tabular_benchmarks.tar.gz && cd fcnet_tabular_benchmarks/

	# 3. Specify problems used in this benchmark.
	#

	$ kurobako solver command python random.py > solvers.json
	$ kurobako solver command python sa.py >> solvers.json
	$ kurobako studies --problems $(cat problems.json) --solvers $(cat solvers.json) \| kurobako run > result.json

	train.batch_size = 10
	train.learning_rate = 0.1