Artemis Documentation

Artemis is a collection of tools that make it easier to run experiments in Python. These include:

• An easy-to-use system for making live plots, to monitor variables in a running experiment.
• A browser-based plotter for displaying live plots.
• A framework for defining experiments and logging their results (text output and figures) so that they can be reviewed later and replicated easily.
• A system for downloading/caching files, to a local directory, so the same code can work on different machines.

To install Artemis, see Artemis on Github

Artemis Experiments Documentation

The Artemis Experiment Framework helps you to keep track of your experiments and their results. It is an alternative to Sacred, with the goal of being more intuitive to use.

For details on the Experiment API, see Experiment API. For a in introduction to the framework, read on…

A Basic Example

Using this module, you can turn your main function into an “Experiment”, which, when run, stores all console output, plots, and computed results to disk (in ~/.artemis/experiments)

Any function that can be called alone with no arguments can be turned into an experiment using the @experiment_function decorator:

from artemis.experiments import experiment_function

@experiment_function
def multiply_3_numbers(a=1, b=2, c=3):
    return a*b*c

This turns the function into an Experiment object, which, in addition to still being a callable function, has methods run(), add_variant(...) and get_variant(). It’s important to give this function a unique name (rather than main(), or something) because this name is used to link the experiment to the records that it has produced.

If we want to run our experiment, and save all text outputs and plots to disk, we can call the run method:

record = multiply_3_numbers.run()

Before we get to reviewing the results, we may want to create a “variant” on this experiment, with a different set of parameters. For this, we can use the add_variant method:

multiply_3_numbers.add_variant('higher-ab', a=4, b=5)

If we want to access this variant later, we can call get_variant:.

ex = multiply_3_numbers.get_variant('higher-ab')

To open up a menu where you can see and run all experiments (and their variants) that have been created we run:

multiply_3_numbers.browse()

This will give us an output that looks something like this:

==================== Experiments ====================
  E#  R#    Name                          All Runs                    Duration         Status           Valid    Result
----  ----  ----------------------------  --------------------------  ---------------  ---------------  -------  --------
   0  0     multiply_3_numbers            2017-08-03 10:34:51.150555  0.0213599205017  Ran Succesfully  Yes      6
   1        multiply_3_numbers.higher-ab  <No Records>                -                -                -        -
-----------------------------------------------------
Enter command or experiment # to run (h for help) >>

This indicates that we have a saved record of our experiment (created when we called multiply_3_numbers.run()), but none of the variant higher-ab. In the UI, we can run this variant by entering run 1:

Enter command or experiment # to run (h for help) >> run 1

After running, we will see the status of our experiments updated:

==================== Experiments ====================
  E#    R#  Name                          All Runs                      Duration  Status           Valid      Result
----  ----  ----------------------------  --------------------------  ----------  ---------------  -------  --------
   0     0  multiply_3_numbers            2017-08-03 10:34:51.150555   0.0213599  Ran Succesfully  Yes             6
   1     0  multiply_3_numbers.higher-ab  2017-08-03 10:38:45.836260   0.0350862  Ran Succesfully  Yes            60
-----------------------------------------------------
Enter command or experiment # to run (h for help) >>

From the UI we have access to a variety of commands for showing and comparing experiments. For example, argtable prints a table comparing the results of the different experiments:

Enter command or experiment # to run (h for help) >> argtable all
    -------------------------------------------------------  ------------------  ---------------  -----------  --------------  ------
                                                             Function            Run Time         Common Args  Different Args  Result
    2017.08.03T10.34.51.150555-multiply_3_numbers            multiply_3_numbers  0.0213599205017  c=3          a=1, b=2        6
    2017.08.03T10.38.45.836260-multiply_3_numbers.higher-ab  multiply_3_numbers  0.0350861549377  c=3          a=4, b=5        60
    -------------------------------------------------------  ------------------  ---------------  -----------  --------------  ------

More Examples

• An example demonstrating Artemis’s Experiment framework on a simple MNIST classification task
• Step-by-step tutorial on using Artemis to organize your Experiments

Experiment API

You can use the UI for most things related to running and viewing the results of experiments. (See Artemis Experiments Documentation for how to do that).

This document shows the methods for interacting programatically with the experiment interface.

Creating Experiments

Experiment Decorators are turn your python functions into experiments. When using decorators, be sure that your experiment function is uniquely named (ie, no other function in your code has the same name). This is important because when results are saved, the name of the function is used to identify what experiment the results belong to.

artemis.experiments.experiment_function(f)

Use this decorator (@experiment_function) on a function that you want to run. e.g.

@experiment_function
def demo_my_experiment(a=1, b=2, c=3):
    ...

This turns your function demo_my_experiment into an experiment. It can still be called as a normal function, but it now has can also be called with the methods of an Experiment object (eg. demo_my_experiment.run()).

artemis.experiments.experiment_root(f)

Use this decorator on a function that you want to build variants off of:

@experiment_root
def demo_my_experiment(a, b=2, c=3):
    ...

The root experiment is not runnable by itself, and will not appear in the list in the browse experiments UI, but you can call demo_my_experiment.add_variant(...) to create runnable variants.

class artemis.experiments.ExperimentFunction(show=<function show_record>, compare=<function compare_experiment_records>, display_function=None, comparison_function=None, one_liner_function=<function sensible_str>, is_root=False)

This is the most general decorator. You can use this to add details on the experiment.

__init__(show=<function show_record>, compare=<function compare_experiment_records>, display_function=None, comparison_function=None, one_liner_function=<function sensible_str>, is_root=False)

Parameters:

• show – A function that is called when you “show” an experiment record in the UI. It takes an experiment record as an argument.
• compare – A function that is called when you “compare” a set of experiment records in the UI.
• display_function – [Deprecated] A function that takes the results (whatever your experiment returns) and displays them.
• comparison_function – [Deprecated] A function that takes an OrderedDict<experiment_name, experiment_return_value>. You can optionally define this function to compare the results of different experiments. You can use call this via the UI with the compare_experiment_results command.
• one_liner_function – A function that takes your results and returns a 1 line string summarizing them.
• is_root – True to make this a root experiment - so that it is not listed to be run itself.

artemis.experiments.capture_created_experiments(*args, **kwds)

A convenient way to cross-breed experiments. If you define experiments in this block, you can capture them for later use (for instance by modifying them). e.g.:

@experiment_function
def add_two_numbers(a=1, b=2):
    return a+b

with capture_created_experiments() as exps:
    add_two_numbers.add_variant(a=2)
    add_two_numbers.add_variant(a=3)

for ex in exps:
    ex.add_variant(b=4)

Return type:Generator[Experiment]

The Experiment

The above decorators return Experiment Objects, which have the following API…

class artemis.experiments.experiments.Experiment(function=None, show=None, compare=None, one_liner_function=None, name=None, is_root=False)

An experiment. In general you should not use this class directly. Use the experiment_function decorator, and create variants using decorated_function.add_variant()

add_config_root_variant(name, **arg_constructors)

Add a config variant which requires additional parametrization. (See add_config_variant)

add_config_variant(name, **arg_constructors)

Add a variant where you redefine the constructor for arguments to the experiment. e.g.

@experiment_function def demo_smooth_out_signal(smoother, signal):

y = [smoother.update(xt) for xt in signal] plt.plot(y) return y

demo_average_sequence.add_config_variant(‘exp_smooth’, averager = lambda decay=0.1: ExponentialMovingAverage(decay))

This creates a variant “exp_smooth” which can be parameterized by a “decay” argument.

Parameters:

• name – Name of the variant
• kwargs – The constructors for the arguments which you’d like to configure.

Returns:

A new experiment.

add_root_variant(variant_name=None, **kwargs)

Add a variant to this experiment, but do NOT register it on the list of experiments. There are two ways you can do this:

# Name the experiment explicitely, then list the named arguments
my_experiment_function.add_root_variant('big_a', a=10000)
assert my_experiment_function.get_name()=='my_experiment_function.big_a'

# Allow the experiment to be named automatically, and just list the named arguments
my_experiment_function.add_root_variant(a=10000)
assert my_experiment_function.get_name()=='my_experiment_function.a=10000'

Parameters:

• variant_name – Optionally, the name of the experiment
• kwargs – The named arguments which will differ from the base experiment.

Returns:

The experiment.

add_variant(variant_name=None, **kwargs)

Add a variant to this experiment, and register it on the list of experiments. There are two ways you can do this:

# Name the experiment explicitely, then list the named arguments
my_experiment_function.add_variant('big_a', a=10000)
assert my_experiment_function.get_name()=='my_experiment_function.big_a'

# Allow the experiment to be named automatically, and just list the named arguments
my_experiment_function.add_variant(a=10000)
assert my_experiment_function.get_name()=='my_experiment_function.a=10000'

Parameters:

• variant_name – Optionally, the name of the experiment
• kwargs – The named arguments which will differ from the base experiment.

Returns:

The experiment.

browse(command=None, catch_errors=False, close_after=False, filterexp=None, filterrec=None, view_mode='full', raise_display_errors=False, run_args=None, keep_record=True, truncate_result_to=100, cache_result_string=False, remove_prefix=None, display_format='nested', **kwargs)

Open up the UI, which allows you to run experiments and view their results.

Parameters:

• command – Optionally, a string command to pass directly to the UI. (e.g. “run 1”)
• catch_errors – Catch errors that arise while running experiments
• close_after – Close after issuing one command.
• filterexp – Filter the experiments with this selection (see help for how to use)
• filterrec – Filter the experiment records with this selection (see help for how to use)
• view_mode – How to view experiments {‘full’, ‘results’} (‘results’ leads to a narrower display).
• raise_display_errors – Raise errors that arise when displaying the table (otherwise just indicate that display failed in table)
• run_args – A dict of named arguments to pass on to Experiment.run
• keep_record – Keep a record of the experiment after running.
• truncate_result_to – An integer, indicating the maximum length of the result string to display.
• cache_result_string – Cache the result string (useful when it takes a very long time to display the results when opening up the menu - often when results are long lists).
• remove_prefix – Remove the common prefix on the experiment ids in the display.
• display_format – How experements and their records are displayed: ‘nested’ or ‘flat’. ‘nested’ might be better for narrow console outputs.

call(*args, **kwargs)

Call the experiment function without running as an experiment. If the experiment is a function, this is the same as just result = my_exp_func(). If it’s defined as a generator, it loops and returns the last result. :return: The last result

copy_variants(other_experiment)

Copy over the variants from another experiment.

Parameters:other_experiment – An Experiment Object

get_all_variants(include_roots=False, include_self=True)

Return a list of variants of this experiment :param include_roots: Include “root” experiments :param include_self: Include this experiment (unless include_roots is false and this this experiment is a root) :return: A list of experiments.

get_args()

Return OrderedDict[str, Any]:  An OrderedDict of arguments to the experiment

get_id()

Returns:A string uniquely identifying this experiment.

get_latest_record(only_completed=False, if_none='skip')

Return the ExperimentRecord from the latest run of this Experiment.

Parameters:

• only_completed – Only search among records of that have run to completion.
• err_if_none – If True, raise an error if no record exists. Otherwise, just return None in this case.

Return ExperimentRecord:  

An ExperimentRecord object

get_records(only_completed=False)

Get all records associated with this experiment.

Parameters:only_completed – Only include records that have run to completion. Returns:A list of ExperimentRecord objects.

get_variant(variant_name=None, **kwargs)

Get a variant on this experiment.

Parameters:

• variant_name (str) – The name of the variant, if it has one
• kwargs – Otherwise, the named arguments which were used to define the variant.

Return Experiment:  

An Experiment object

get_variant_records(only_completed=False, only_last=False, flat=False)

Get the collection of records associated with all variants of this Experiment.

Parameters:

• only_completed – Only search among records of that have run to completion.
• only_last – Just return the most recent record.
• flat – Just return a list of records

Returns:

if not flat (default) An OrderedDict<experiment_id: ExperimentRecord>. otherwise, if flat: a list<ExperimentRecord>

has_record(completed=True, valid=True)

Return true if the experiment has a record, otherwise false. :param completed: Check that the record is completed. :param valid: Check that the record is valid (arguments match current experiment arguments) :return: True/False

run(print_to_console=True, show_figs=None, test_mode=None, keep_record=None, raise_exceptions=True, display_results=False, notes=(), **experiment_record_kwargs)

Run the experiment, and return the ExperimentRecord that is generated.

Parameters:

• print_to_console – Print to console (as well as logging to file)
• show_figs – Show figures (as well as saving to file)
• test_mode – Run in “test_mode”. This sets the global “test_mode” flag when running the experiment. This flag can be used to, for example, shorten a training session to verify that the code runs. Can be: True: Run in test mode False: Don’t run in test mode: None: Keep the current state of the global “is_test_mode()” flag.
• keep_record – Keep the folder that results are saved into. True: Results are saved into a folder False: Results folder is deleted at the end. None: If “test_mode” is true, then delete results at end, otherwise save them.
• raise_exceptions – True to raise any exception that occurs when running the experiment. False to catch it, print the error, and move on.
• experiment_record_kwargs – Passed to the “record_experiment” context.

Returns:

The ExperimentRecord object, if keep_record is true, otherwise None

The Experiment Record

When you run an Experiment, a folder is created in which the stdout, results, figures, and other info are stored. The ExperimentRecord object provides an API for accessing the contents of this folder.

class artemis.experiments.experiment_record.ExperimentRecord(experiment_directory)

A Record of a run of an Experiment. This object allows you to access data stored in the directory that was created for that run of the experiment. Experiment Records are stored in ~/artemis/experiments/.

args_valid(last_run_args=None, current_args=None)

Parameters:

• last_run_args (Optional[OrderedDict]) – The arguments from the last run
• current_args (Optional[OrderedDict]) – The arguments from the current experiment in code

Returns:

True if the experiment arguments have not changed False if they have changed None if it cannot be determined because arguments are not hashable objects.

delete()

Delete this experiment record from disk.

get_args()

Get the arguments with which this record was run. :return: An OrderedDict((arg_name -> arg_value))

get_dir()

Returns:The directory associated with this experiment record.

get_error_trace()

Get the error trace, or return None if there is no error trace. :return:

get_experiment()

Load the experiment associated with this record. Note that this will raise an ExperimentNotFoundError if the experiment has not been imported. :return: An Experiment object

get_figure_locs(include_directory=True)

Return a list of the paths of the figures saved in the experiment. :param include_directory: If True, return the full path. :return: A list of string file paths.

get_id()

Get the id of this experiment record. Generally in format ‘<datetime>-<experiment_name>’ :return:

get_log()

Returns:The stdout generated during the run of this experiment.

get_result(err_if_none=True)

Unpickle and return the “return value” of the experiment. :param err_if_none: If there is no saved return value, throw an exception if err_is_none, else just return None. :return: The return value from the experiment.

get_runtime()

Return datetime.timedelta:  A timedelta object

has_result()

Returns:True if this record has a saved result.

info

Returns:An ExperimentRecordInfo object, containing info about the experiment (name, runtime, etc)

list_files(full_path=False)

List files in experiment directory, relative to root. :param full_path: If true, list file with the full local path :return: A list of strings indicating the file paths.

load_figures()

Returns:A list of matplotlib figures generated in the experiment. The figures will not be drawn yet, so you will have to call plt.show() to draw them or plt.draw() to draw them.

open_file(filename, *args, **kwargs)

Open a file within the experiment record folder. Example Usage:

with record.open_file(‘filename.txt’) as f:

txt = f.read()

Parameters:

• filename – Path within experiment directory (it can include subdirectories)
• kwargs (args,) – Forwarded to python’s “open” function

Returns:

A file object

show_figures(hang=False)

Show all figures that were saved during the run of the experiment. :param hang: If True, and figures were saved matplotlib figures, hang execution until they are closed.

Artemis Plotting

Live Plots with dbplot

dbplot is an easy way to create a live plot of your data.

For example, to create live updating plots of a random grid:

from artemis.plotting.db_plotting import dbplot
import numpy as np

for _ in xrange(50):
    dbplot(np.random.randn(20, 10), 'random data')

A plot will come up showing the random data.

from artemis.plotting.db_plotting import dbplot
import numpy as np

for _ in xrange(50):
    dbplot(np.random.randn(20, 10), 'random line data', plot_type='line')

You can include multiple plots:

from artemis.plotting.db_plotting import dbplot
import numpy as np

for _ in xrange(50):
    dbplot(np.random.randn(20, 2), 'random line data', plot_type='line')
    dbplot(np.random.randn(10, 10), 'random grid data')

If you plot many things, you may want to “hold” your plots, so that they all update together. This speeds up the rate of plotting:

from artemis.plotting.db_plotting import dbplot, hold_dbplots
import numpy as np

for _ in xrange(50):
    with hold_dbplots():
        dbplot(np.random.randn(20, 2), 'random line data', plot_type='line')
        dbplot(np.random.randn(10, 10), 'random grid data')
        dbplot(np.random.randn(4), 'random line history')
        dbplot(np.random.rand(3), 'random bars', plot_type='bar')
        dbplot([np.random.rand(20, 20), np.random.randn(20, 16, 3)], 'multi image')

dbplot documentation

artemis.plotting.db_plotting.dbplot(data, name=None, plot_type=None, axis=None, plot_mode='live', draw_now=True, hang=False, title=None, fig=None, xlabel=None, ylabel=None, draw_every=None, layout=None, legend=None, grid=False, wait_for_display_sec=0, cornertext=None, reset_color_cycle=False)

Plot arbitrary data and continue execution. This program tries to figure out what type of plot to use.

Parameters:

• data – Any data. Hopefully, we at dbplot will be able to figure out a plot for it.
• name – A name uniquely identifying this plot.
• plot_type – A specialized constructor to be used the first time when plotting. You can also pass certain string to give hints as to what kind of plot you want (can resolve cases where the given data could be plotted in multiple ways): ‘line’: Plots a line plot ‘img’: An image plot ‘colour’: A colour image plot ‘pic’: A picture (no scale bars, axis labels, etc).
• axis – A string identifying which axis to plot on. By default, it is the same as “name”. Only use this argument if you indend to make multiple dbplots share the same axis.
• plot_mode – Influences how the data should be used to choose the plot type: ‘live’: Best for ‘live’ plots that you intend to update as new data arrives ‘static’: Best for ‘static’ plots, that you do not intend to update ‘image’: Try to represent the plot as an image
• draw_now – Draw the plot now (you may choose false if you’re going to add another plot immediately after and don’t want have to draw this one again.
• hang – Hang on the plot (wait for it to be closed before continuing)
• title – Title of the plot (will default to name if not included)
• fig – Name of the figure - use this when you want to create multiple figures.
• grid – Turn the grid on
• wait_for_display_sec – In server mode, you can choose to wait maximally wait_for_display_sec seconds before this call returns. In case plotting is finished earlier, the call returns earlier. Setting wait_for_display_sec to a negative number will cause the call to block until the plot has been displayed.

Plotting Demos

• A demo of showing how to make various kinds of live updating plots.
• A demo repo showing how to use Artemis from your code
• A guide on using Artemis for remote plotting

Browser-Plotting

After installing, you should have a file ~/.artemisrc.

To use the web backend, edit the backend field to matplotlib-web.

To try it you can run the commands described above for dbplot.