Artemis Experiments Documentation¶
The Artemis Experiment Framework helps you to keep track of your experiments and their results. It is an alternative to Sacred, with the goal of being more intuitive to use.
For details on the Experiment API, see Experiment API. For a in introduction to the framework, read on…
A Basic Example¶
Using this module, you can turn your main function into an “Experiment”, which, when run, stores all console output, plots, and computed results to disk (in ~/.artemis/experiments)
Any function that can be called alone with no arguments can be turned into an experiment using the @experiment_function decorator:
from artemis.experiments import experiment_function
@experiment_function
def multiply_3_numbers(a=1, b=2, c=3):
return a*b*c
This turns the function into an Experiment object, which, in addition to still being a callable function, has methods run()
, add_variant(...)
and get_variant()
. It’s important to give this function a unique name (rather than main()
, or something) because this name is used to link the experiment to the records that it has produced.
If we want to run our experiment, and save all text outputs and plots to disk, we can call the run
method:
record = multiply_3_numbers.run()
Before we get to reviewing the results, we may want to create a “variant” on this experiment, with a different set of parameters. For this, we can use the add_variant
method:
multiply_3_numbers.add_variant('higher-ab', a=4, b=5)
If we want to access this variant later, we can call get_variant
:.
ex = multiply_3_numbers.get_variant('higher-ab')
To open up a menu where you can see and run all experiments (and their variants) that have been created we run:
multiply_3_numbers.browse()
This will give us an output that looks something like this:
==================== Experiments ====================
E# R# Name All Runs Duration Status Valid Result
---- ---- ---------------------------- -------------------------- --------------- --------------- ------- --------
0 0 multiply_3_numbers 2017-08-03 10:34:51.150555 0.0213599205017 Ran Succesfully Yes 6
1 multiply_3_numbers.higher-ab <No Records> - - - -
-----------------------------------------------------
Enter command or experiment # to run (h for help) >>
This indicates that we have a saved record of our experiment (created when we called multiply_3_numbers.run()
), but
none of the variant higher-ab
. In the UI, we can run this variant by entering run 1
:
Enter command or experiment # to run (h for help) >> run 1
After running, we will see the status of our experiments updated:
==================== Experiments ====================
E# R# Name All Runs Duration Status Valid Result
---- ---- ---------------------------- -------------------------- ---------- --------------- ------- --------
0 0 multiply_3_numbers 2017-08-03 10:34:51.150555 0.0213599 Ran Succesfully Yes 6
1 0 multiply_3_numbers.higher-ab 2017-08-03 10:38:45.836260 0.0350862 Ran Succesfully Yes 60
-----------------------------------------------------
Enter command or experiment # to run (h for help) >>
From the UI we have access to a variety of commands for showing and comparing experiments. For example, argtable prints a table comparing the results of the different experiments:
Enter command or experiment # to run (h for help) >> argtable all
------------------------------------------------------- ------------------ --------------- ----------- -------------- ------
Function Run Time Common Args Different Args Result
2017.08.03T10.34.51.150555-multiply_3_numbers multiply_3_numbers 0.0213599205017 c=3 a=1, b=2 6
2017.08.03T10.38.45.836260-multiply_3_numbers.higher-ab multiply_3_numbers 0.0350861549377 c=3 a=4, b=5 60
------------------------------------------------------- ------------------ --------------- ----------- -------------- ------