Experiment API

You can use the UI for most things related to running and viewing the results of experiments. (See Artemis Experiments Documentation for how to do that).

This document shows the methods for interacting programatically with the experiment interface.

Creating Experiments

Experiment Decorators are turn your python functions into experiments. When using decorators, be sure that your experiment function is uniquely named (ie, no other function in your code has the same name). This is important because when results are saved, the name of the function is used to identify what experiment the results belong to.

artemis.experiments.experiment_function(f)

Use this decorator (@experiment_function) on a function that you want to run. e.g.

@experiment_function
def demo_my_experiment(a=1, b=2, c=3):
    ...

This turns your function demo_my_experiment into an experiment. It can still be called as a normal function, but it now has can also be called with the methods of an Experiment object (eg. demo_my_experiment.run()).

artemis.experiments.experiment_root(f)

Use this decorator on a function that you want to build variants off of:

@experiment_root
def demo_my_experiment(a, b=2, c=3):
    ...

The root experiment is not runnable by itself, and will not appear in the list in the browse experiments UI, but you can call demo_my_experiment.add_variant(...) to create runnable variants.

class artemis.experiments.ExperimentFunction(display_function=None, comparison_function=None, one_liner_function=None, is_root=False)

This is the most general decorator. You can use this to add details on the experiment.

__init__(display_function=None, comparison_function=None, one_liner_function=None, is_root=False)

Parameters:

• display_function – A function that takes the results (whatever your experiment returns) and displays them.
• comparison_function – A function that takes an OrderedDict<experiment_name, experiment_return_value>. You can optionally define this function to compare the results of different experiments. You can use call this via the UI with the compare_experiment_results command.
• one_liner_function – A function that takes your results and returns a 1 line string summarizing them.
• is_root – True to make this a root experiment - so that it is not listed to be run itself.

artemis.experiments.capture_created_experiments(*args, **kwds)

A convenient way to cross-breed experiments. If you define experiments in this block, you can capture them for later use (for instance by modifying them). e.g.:

@experiment_function
def add_two_numbers(a=1, b=2):
    return a+b

with capture_created_experiments() as exps:
    add_two_numbers.add_variant(a=2)
    add_two_numbers.add_variant(a=3)

for ex in exps:
    ex.add_variant(b=4)

Return type:Generator[Experiment]

The Experiment

The above decorators return Experiment Objects, which have the following API…

class artemis.experiments.experiments.Experiment(function=None, display_function=None, comparison_function=None, one_liner_function=None, name=None, is_root=False)

An experiment. In general you should not use this class directly. Use the experiment_function decorator, and create variants using decorated_function.add_variant()

add_root_variant(variant_name=None, **kwargs)

Add a variant to this experiment, but do NOT register it on the list of experiments. There are two ways you can do this:

# Name the experiment explicitely, then list the named arguments
my_experiment_function.add_root_variant('big_a', a=10000)
assert my_experiment_function.get_name()=='my_experiment_function.big_a'

# Allow the experiment to be named automatically, and just list the named arguments
my_experiment_function.add_root_variant(a=10000)
assert my_experiment_function.get_name()=='my_experiment_function.a=10000'

Parameters:

• variant_name – Optionally, the name of the experiment
• kwargs – The named arguments which will differ from the base experiment.

Returns:

The experiment.

add_variant(variant_name=None, **kwargs)

Add a variant to this experiment, and register it on the list of experiments. There are two ways you can do this:

# Name the experiment explicitely, then list the named arguments
my_experiment_function.add_variant('big_a', a=10000)
assert my_experiment_function.get_name()=='my_experiment_function.big_a'

# Allow the experiment to be named automatically, and just list the named arguments
my_experiment_function.add_variant(a=10000)
assert my_experiment_function.get_name()=='my_experiment_function.a=10000'

Parameters:

• variant_name – Optionally, the name of the experiment
• kwargs – The named arguments which will differ from the base experiment.

Returns:

The experiment.

browse(command=None, catch_errors=False, close_after=False, just_last_record=False, view_mode='full', raise_display_errors=False, run_args=None, keep_record=True, truncate_result_to=100, cache_result_string=False, **kwargs)

Open up the UI, which allows you to run experiments and view their results.

Parameters:

• command – Optionally, a string command to pass directly to the UI. (e.g. “run 1”)
• catch_errors – Catch errors that arise while running experiments
• close_after – Close after issuing one command.
• just_last_record – Only show the most recent record for each experiment.
• view_mode – How to view experiments {‘full’, ‘results’} (‘results’ leads to a narrower display).
• raise_display_errors – Raise errors that arise when displaying the table (otherwise just indicate that display failed in table)
• run_args – A dict of named arguments to pass on to Experiment.run
• keep_record – Keep a record of the experiment after running.
• truncate_result_to – An integer, indicating the maximum length of the result string to display.
• cache_result_string – Cache the result string (useful when it takes a very long time to display the results when opening up the menu - often when results are long lists).

get_all_variants(include_roots=False, include_self=True)

Return a list of variants of this experiment :param include_roots: Include “root” experiments :param include_self: Include this experiment (unless include_roots is false and this this experiment is a root) :return: A list of experiments.

get_args()

Returns:A dictionary of arguments to the experiment

get_id()

Returns:A string uniquely identifying this experiment.

get_latest_record(only_completed=False, err_if_none=True)

Return the ExperimentRecord from the latest run of this Experiment.

Parameters:

• only_completed – Only search among records of that have run to completion.
• err_if_none – If True, raise an error if no record exists. Otherwise, just return None in this case.

Returns:

An ExperimentRecord object

get_records(only_completed=False)

Get all records associated with this experiment.

Parameters:only_completed – Only include records that have run to completion. Returns:A list of ExperimentRecord objects.

get_variant(variant_name=None, **kwargs)

Get a variant on this experiment.

Parameters:

• variant_name – The name of the variant, if it has one
• kwargs – Otherwise, the named arguments which were used to define the variant.

Returns:

An Experiment object

get_variant_records(only_completed=False, only_last=False, flat=False)

Get the collection of records associated with all variants of this Experiment.

Parameters:

• only_completed – Only search among records of that have run to completion.
• only_last – Just return the most recent record.
• flat – Just return a list of records

Returns:

if not flat (default) An OrderedDict<experiment_id: ExperimentRecord>. otherwise, if flat: a list<ExperimentRecord>

has_record(completed=True, valid=True)

Return true if the experiment has a record, otherwise false. :param completed: Check that the record is completed. :param valid: Check that the record is valid (arguments match current experiment arguments) :return: True/False

run(print_to_console=True, show_figs=None, test_mode=None, keep_record=None, raise_exceptions=True, display_results=True, **experiment_record_kwargs)

Run the experiment, and return the ExperimentRecord that is generated.

Parameters:

• print_to_console – Print to console (as well as logging to file)
• show_figs – Show figures (as well as saving to file)
• test_mode – Run in “test_mode”. This sets the global “test_mode” flag when running the experiment. This flag can be used to, for example, shorten a training session to verify that the code runs. Can be: True: Run in test mode False: Don’t run in test mode: None: Keep the current state of the global “is_test_mode()” flag.
• keep_record – Keep the folder that results are saved into. True: Results are saved into a folder False: Results folder is deleted at the end. None: If “test_mode” is true, then delete results at end, otherwise save them.
• raise_exceptions – True to raise any exception that occurs when running the experiment. False to catch it, print the error, and move on.
• experiment_record_kwargs – Passed to the “record_experiment” context.

Returns:

The ExperimentRecord object, if keep_record is true, otherwise None

The Experiment Record

When you run an Experiment, a folder is created in which the stdout, results, figures, and other info are stored. The ExperimentRecord object provides an API for accessing the contents of this folder.

class artemis.experiments.experiment_record.ExperimentRecord(experiment_directory)

A Record of a run of an Experiment. This object allows you to access data stored in the directory that was created for that run of the experiment. Experiment Records are stored in ~/artemis/experiments/.

args_valid(last_run_args=None, current_args=None)

Returns:True if the experiment arguments have not changed False if they have changed None if it cannot be determined because arguments are not hashable objects.

delete()

Delete this experiment record from disk.

get_args()

Get the arguments with which this record was run. :return: An OrderedDict((arg_name -> arg_value))

get_dir()

Returns:The directory associated with this experiment record.

get_error_trace()

Get the error trace, or return None if there is no error trace. :return:

get_experiment()

Load the experiment associated with this record. Note that this will raise an ExperimentNotFoundError if the experiment has not been imported. :return: An Experiment object

get_figure_locs(include_directory=True)

Return a list of the paths of the figures saved in the experiment. :param include_directory: If True, return the full path. :return: A list of string file paths.

get_id()

Get the id of this experiment record. Generally in format ‘<datetime>-<experiment_name>’ :return:

get_log()

Returns:The stdout generated during the run of this experiment.

get_result(err_if_none=True)

Unpickle and return the “return value” of the experiment. :param err_if_none: If there is no saved return value, throw an exception if err_is_none, else just return None. :return: The return value from the experiment.

has_result()

Returns:True if this record has a saved result.

info

Returns:An ExperimentRecordInfo object, containing info about the experiment (name, runtime, etc)

list_files(full_path=False)

List files in experiment directory, relative to root. :param full_path: If true, list file with the full local path :return: A list of strings indicating the file paths.

load_figures()

Returns:A list of matplotlib figures generated in the experiment. The figures will not be drawn yet, so you will have to call plt.show() to draw them or plt.draw() to draw them.

open_file(filename, *args, **kwargs)

Open a file within the experiment record folder. Example Usage:

with record.open_file(‘filename.txt’) as f:

txt = f.read()

Parameters:

• filename – Path within experiment directory (it can include subdirectories)
• kwargs (args,) – Forwarded to python’s “open” function

Returns:

A file object

show_figures(hang=False)

Show all figures that were saved during the run of the experiment. :param hang: If True, and figures were saved matplotlib figures, hang execution until they are closed.