dowel package

Logger module.

This module instantiates a global logger singleton.

class dowel.Histogram

Bases: numpy.ndarray

A dowel.logger input representing a histogram of raw data.

This is implemented as a typed view of a numpy array. It will accept input that numpy.asarray will.

See https://docs.scipy.org/doc/numpy/user/basics.subclassing.html for details on implementation.

class dowel.Logger

Bases: object

This is the class that handles logging.

add_output(output)

Add a new output to the logger.

All data that is compatible with this output will be sent there.

Parameters:output – An instantiation of a LogOutput subclass to be added.

disable_warnings()

Disable logger warnings for testing.

dump_all(step=None)

Dump all outputs connected to the logger.

Parameters:step – The current run step.

dump_output_type(output_type, step=None)

Dump all outputs of the given type.

Parameters:

• output_type – A LogOutput subclass type to be dumped.
• step – The current run step.

has_output_type(output_type)

Check to see if a given logger output is attached to the logger.

Parameters:output_type – A LogOutput subclass type to be checked for.

log(data)

Magic method that takes in all different types of input.

This method is the main API for the logger. Any data to be logged goes through this method.

Any data sent to this method is sent to all outputs that accept its type (defined in the types_accepted property).

Parameters:data – Data to be logged. This can be any type specified in the types_accepted property of any of the logger outputs.

pop_prefix()

Pop prefix from prefix stack.

prefix(prefix)

Add a prefix to the logger.

This allows text output to be prepended with a given stack of prefixes.

Example: with logger.prefix(‘prefix: ‘):

logger.log(‘test_string’) # this will have the prefix

logger.log(‘test_string2’) # this will not have the prefix

Parameters:prefix – The prefix string to be logged.

push_prefix(prefix)

Add prefix to prefix stack.

Parameters:prefix – The prefix string to be logged.

remove_all()

Remove all outputs that have been added to this logger.

remove_output_type(output_type)

Remove all outputs of a given type.

Parameters:output_type – A LogOutput subclass type to be removed.

reset_output(output)

Removes, then re-adds a given output to the logger.

Parameters:output – An instantiation of a LogOutput subclass to be added.

class dowel.CsvOutput(file_name)

Bases: dowel.simple_outputs.FileOutput

CSV file output for logger.

Parameters:file_name – The file this output should log to.

record(data, prefix='')

Log tabular data to CSV.

types_accepted

Accept TabularInput objects only.

class dowel.StdOutput(with_timestamp=True)

Bases: dowel.logger.LogOutput

Standard console output for the logger.

Parameters:with_timestamp – Whether to log a timestamp before non-tabular data.

dump(step=None)

Flush data to standard output stream.

record(data, prefix='')

Log data to console.

types_accepted

Accept str and TabularInput objects.

class dowel.TextOutput(file_name, with_timestamp=True)

Bases: dowel.simple_outputs.FileOutput

Text file output for logger.

Parameters:

• file_name – The file this output should log to.
• with_timestamp – Whether to log a timestamp before the data.

record(data, prefix='')

Log data to text file.

types_accepted

Accept str objects only.

class dowel.LogOutput

Bases: abc.ABC

Abstract class for Logger Outputs.

close()

Close any files used by the output.

dump(step=None)

Dump the contents of this output.

Parameters:step – The current run step.

record(data, prefix='')

Pass logger data to this output.

Parameters:

• data – The data to be logged by the output.
• prefix – A prefix placed before a log entry in text outputs.

types_accepted

Pass these types to this logger output.

The types in this tuple will be accepted by this output.

Returns:A tuple containing all valid input types.

exception dowel.LoggerWarning

Bases: UserWarning

Warning class for the Logger.

class dowel.TabularInput

Bases: object

This class allows the user to create tables for easy display.

TabularInput may be passed to the logger via its log() method.

as_dict

Return a dictionary of the tabular items.

as_primitive_dict

Return the dictionary, excluding all nonprimitive types.

clear()

Clear the tabular.

disable_warnings()

Disable logger warnings for testing.

mark(key)

Mark key as recorded.

mark_all()

Mark all keys.

mark_str()

Mark keys in the primitive dict.

pop_prefix()

Pop prefix that was appended to the printed table.

prefix(prefix)

Handle pushing and popping of a tabular prefix.

Can be used in the following way:

with tabular.prefix(‘your_prefix_’):

# your code tabular.record(key, val)

Parameters:prefix – The string prefix to be prepended to logs.

push_prefix(prefix)

Push prefix to be appended before printed table.

Parameters:prefix – The string prefix to be prepended to logs.

record(key, val)

Save key/value entries for the table.

Parameters:

• key – String key corresponding to the value.
• val – Value that is to be stored in the table.

record_misc_stat(key, values, placement='back')

Record statistics of an array.

Parameters:

• key – String key corresponding to the values.
• values – Array of values to be analyzed.
• placement – Whether to put the prefix in front or in the back.

class dowel.TensorBoardOutput(log_dir, x_axis=None, additional_x_axes=None, flush_secs=120, histogram_samples=1000.0)

Bases: dowel.logger.LogOutput

TensorBoard output for logger.

Parameters:

• log_dir (str) – The save location of the tensorboard event files.
• x_axis (str) – The name of data used as x-axis for scalar tabular. If None, x-axis will be the number of dump() is called.
• additional_x_axes (list[str]) – Names of data to used be as additional x-axes.
• flush_secs (int) – How often, in seconds, to flush the added summaries and events to disk.
• histogram_samples (int) – Number of samples to generate when logging random distribution.

close()

Flush all the events to disk and close the file.

dump(step=None)

Flush summary writer to disk.

record(data, prefix='')

Add data to tensorboard summary.

Parameters:

• data – The data to be logged by the output.
• prefix (str) – A prefix placed before a log entry in text outputs.

types_accepted

Return the types that the logger may pass to this output.

Submodules

dowel.csv_output module

A dowel.logger.LogOutput for CSV files.

class dowel.csv_output.CsvOutput(file_name)

Bases: dowel.simple_outputs.FileOutput

CSV file output for logger.

Parameters:file_name – The file this output should log to.

record(data, prefix='')

Log tabular data to CSV.

types_accepted

Accept TabularInput objects only.

dowel.histogram module

Histogram logger input.

class dowel.histogram.Histogram

Bases: numpy.ndarray

A dowel.logger input representing a histogram of raw data.

This is implemented as a typed view of a numpy array. It will accept input that numpy.asarray will.

See https://docs.scipy.org/doc/numpy/user/basics.subclassing.html for details on implementation.

dowel.logger module

Logging facility.

It takes in many different types of input and directs them to the correct output.

The logger has 4 major steps:

1. Inputs, such as a simple string or something more complicated like TabularInput, are passed to the log() method of an instantiated Logger.

2. The Logger class checks for any outputs that have been added to it, and calls the record() method of any outputs that accept the type of input.

3. The output (a subclass of LogOutput) receives the input via its record() method and handles it in whatever way is expected.

4. (only in some cases) The dump method is used to dump the output to file. It is necessary for some LogOutput subclasses, like TensorBoardOutput.

# Here’s a demonstration of dowel:

from dowel import logger

logger

# Let’s add an output to the logger. We want to log to the console, so we’ll # add a StdOutput.

from dowel import StdOutput logger.add_output(StdOutput())

+——+ +———+ |logger+------>StdOutput| +——+ +———+

# Great! Now we can start logging text.

logger.log(‘Hello dowel’)

# This will go straight to the console as ‘Hello dowel’

+——+ +———+ |logger+---'Hello dowel'--->StdOutput| +——+ +———+

# Let’s try adding another output.

from dowel import TextOutput logger.add_output(TextOutput(‘log_folder/log.txt’))

+——>StdOutput|

+——+ +———+ |logger| +——+ +———-+

+——>TextOutput|

# And another output.

from dowel import CsvOutput logger.add_output(CsvOutput(‘log_folder/table.csv’))

+——>StdOutput| | +———+ |

+——+ +———-+ |logger+------>TextOutput| +——+ +———-+

+———+

+——>CsvOutput|

# The logger will record anything passed to logger.log to all outputs that # accept its type.

logger.log(‘test’)

+—‘test’—>StdOutput| | +———+ |

+——+ +———-+ |logger+---'test'--->TextOutput| +——+ +———-+

+———+

+—–!!—–>CsvOutput|

# !! Note that the logger knows not to send CsvOutput the string ‘test’ # Similarly, more complex objects like tf.tensor won’t be sent to (for # example) TextOutput. # This behavior is defined in each output’s types_accepted property

# Here’s a more complex example. # TabularInput, instantiated for you as the tabular, can log key/value pairs.

from dowel import tabular tabular.record(‘key’, 72) tabular.record(‘foo’, ‘bar’) logger.log(tabular)

+—tabular—>StdOutput| | +———+ |

+——+ +———-+ |logger+---tabular--->TextOutput| +——+ +———-+

+———+

+—tabular—>CsvOutput|

# Note that LogOutputs which consume TabularInputs must call # TabularInput.mark() on each key they log. This helps the logger detect when # tabular data is not logged.

# Console Output: — — key 72 foo bar — —

# Feel free to add your own inputs and outputs to the logger!

class dowel.logger.LogOutput

Bases: abc.ABC

Abstract class for Logger Outputs.

close()

Close any files used by the output.

dump(step=None)

Dump the contents of this output.

Parameters:step – The current run step.

record(data, prefix='')

Pass logger data to this output.

Parameters:

• data – The data to be logged by the output.
• prefix – A prefix placed before a log entry in text outputs.

types_accepted

Pass these types to this logger output.

The types in this tuple will be accepted by this output.

Returns:A tuple containing all valid input types.

class dowel.logger.Logger

Bases: object

This is the class that handles logging.

add_output(output)

Add a new output to the logger.

All data that is compatible with this output will be sent there.

Parameters:output – An instantiation of a LogOutput subclass to be added.

disable_warnings()

Disable logger warnings for testing.

dump_all(step=None)

Dump all outputs connected to the logger.

Parameters:step – The current run step.

dump_output_type(output_type, step=None)

Dump all outputs of the given type.

Parameters:

• output_type – A LogOutput subclass type to be dumped.
• step – The current run step.

has_output_type(output_type)

Check to see if a given logger output is attached to the logger.

Parameters:output_type – A LogOutput subclass type to be checked for.

log(data)

Magic method that takes in all different types of input.

This method is the main API for the logger. Any data to be logged goes through this method.

Any data sent to this method is sent to all outputs that accept its type (defined in the types_accepted property).

Parameters:data – Data to be logged. This can be any type specified in the types_accepted property of any of the logger outputs.

pop_prefix()

Pop prefix from prefix stack.

prefix(prefix)

Add a prefix to the logger.

This allows text output to be prepended with a given stack of prefixes.

Example: with logger.prefix(‘prefix: ‘):

logger.log(‘test_string’) # this will have the prefix

logger.log(‘test_string2’) # this will not have the prefix

Parameters:prefix – The prefix string to be logged.

push_prefix(prefix)

Add prefix to prefix stack.

Parameters:prefix – The prefix string to be logged.

remove_all()

Remove all outputs that have been added to this logger.

remove_output_type(output_type)

Remove all outputs of a given type.

Parameters:output_type – A LogOutput subclass type to be removed.

reset_output(output)

Removes, then re-adds a given output to the logger.

Parameters:output – An instantiation of a LogOutput subclass to be added.

exception dowel.logger.LoggerWarning

Bases: UserWarning

Warning class for the Logger.

dowel.simple_outputs module

Contains the output classes for the logger.

Each class is sent logger data and handles it itself.

class dowel.simple_outputs.FileOutput(file_name, mode='w')

Bases: dowel.logger.LogOutput

File output abstract class for logger.

Parameters:

• file_name – The file this output should log to.
• mode – File open mode (‘a’, ‘w’, etc).

close()

Close any files used by the output.

dump(step=None)

Flush data to log file.

class dowel.simple_outputs.StdOutput(with_timestamp=True)

Bases: dowel.logger.LogOutput

Standard console output for the logger.

Parameters:with_timestamp – Whether to log a timestamp before non-tabular data.

dump(step=None)

Flush data to standard output stream.

record(data, prefix='')

Log data to console.

types_accepted

Accept str and TabularInput objects.

class dowel.simple_outputs.TextOutput(file_name, with_timestamp=True)

Bases: dowel.simple_outputs.FileOutput

Text file output for logger.

Parameters:

• file_name – The file this output should log to.
• with_timestamp – Whether to log a timestamp before the data.

record(data, prefix='')

Log data to text file.

types_accepted

Accept str objects only.

dowel.tabular_input module

A dowel.logger input for tabular (key-value) data.

class dowel.tabular_input.TabularInput

Bases: object

This class allows the user to create tables for easy display.

TabularInput may be passed to the logger via its log() method.

as_dict

Return a dictionary of the tabular items.

as_primitive_dict

Return the dictionary, excluding all nonprimitive types.

clear()

Clear the tabular.

disable_warnings()

Disable logger warnings for testing.

mark(key)

Mark key as recorded.

mark_all()

Mark all keys.

mark_str()

Mark keys in the primitive dict.

pop_prefix()

Pop prefix that was appended to the printed table.

prefix(prefix)

Handle pushing and popping of a tabular prefix.

Can be used in the following way:

with tabular.prefix(‘your_prefix_’):

# your code tabular.record(key, val)

Parameters:prefix – The string prefix to be prepended to logs.

push_prefix(prefix)

Push prefix to be appended before printed table.

Parameters:prefix – The string prefix to be prepended to logs.

record(key, val)

Save key/value entries for the table.

Parameters:

• key – String key corresponding to the value.
• val – Value that is to be stored in the table.

record_misc_stat(key, values, placement='back')

Record statistics of an array.

Parameters:

• key – String key corresponding to the values.
• values – Array of values to be analyzed.
• placement – Whether to put the prefix in front or in the back.

exception dowel.tabular_input.TabularInputWarning

Bases: UserWarning

Warning class for the TabularInput.

dowel.tensor_board_output module

A dowel.logger.LogOutput for tensorboard.

It receives the input data stream from dowel.logger, then add them to tensorboard summary operations through tensorboardX.

Note

Neither TensorboardX nor TensorBoard supports log parametric distributions. We add this feature by sampling data from a tfp.distributions.Distribution object.

exception dowel.tensor_board_output.NonexistentAxesWarning

Bases: dowel.logger.LoggerWarning

Raise when the specified x axes do not exist in the tabular.

class dowel.tensor_board_output.TensorBoardOutput(log_dir, x_axis=None, additional_x_axes=None, flush_secs=120, histogram_samples=1000.0)

Bases: dowel.logger.LogOutput

TensorBoard output for logger.

Parameters:

• log_dir (str) – The save location of the tensorboard event files.
• x_axis (str) – The name of data used as x-axis for scalar tabular. If None, x-axis will be the number of dump() is called.
• additional_x_axes (list[str]) – Names of data to used be as additional x-axes.
• flush_secs (int) – How often, in seconds, to flush the added summaries and events to disk.
• histogram_samples (int) – Number of samples to generate when logging random distribution.

close()

Flush all the events to disk and close the file.

dump(step=None)

Flush summary writer to disk.

record(data, prefix='')

Add data to tensorboard summary.

Parameters:

• data – The data to be logged by the output.
• prefix (str) – A prefix placed before a log entry in text outputs.

types_accepted

Return the types that the logger may pass to this output.

dowel.utils module

Utilities for console outputs.

dowel.utils.colorize(string, color, bold=False, highlight=False)

Colorize the string for console output.

dowel.utils.mkdir_p(path)

Create a directory with path.