Hyperparameters

Abstract Hyperparameter Class

class ftw.tf.hyperparameters.base.Hyperparameter

Bases: object

Metaclass that describes an API for hyperparameters.

Has one private attribute:
  • _name: Name of the Hyperparameter.

Offers the following public methods, which must be implemented by classes inheriting from Hyperparameter:

  • get(): returns the value of the hyperparameter
  • set(value): sets the value of the hyperparameter
  • perturb(): perturbs the value of the hyperparameter.
get()

Returns the value of the hyperparameter variable.

perturb()

Perturbs/Mutates the value of the hyperparameter variable.

set(value)

Sets the value of the hyperparameter variable.

Float Hyperparameter

class ftw.tf.hyperparameters.ftw_hp.FloatHyperparameter(init_value_or_range: Union[float, Tuple[float, float]], perturb_probability: float = 1.0, perturb_max_pct_change: float = 0.2, dtype=tf.float32, name: str = 'float_hyperparameter')

Bases: ftw.tf.hyperparameters.base.Hyperparameter

Hyperparameter with a scalar float value.

Can be initialized either with a concrete scalar value or a tuple (min, max) indicating a range from which to draw a random sample. More specifically, the sample is drawn from a log-uniform distribution defined over this range.

Contains a tf.Variable that stores the hyperparameter value. Consequently, this module is well suited to be used as a hyperparameter variable inside a tf.function, e.g. in population-based training.

__init__(init_value_or_range: Union[float, Tuple[float, float]], perturb_probability: float = 1.0, perturb_max_pct_change: float = 0.2, dtype=tf.float32, name: str = 'float_hyperparameter')
Args:
init_value_or_range: A scalar value of type int for initialization with an exact value,
or a tuple (min, max), where min, max are of type int for initialization by drawing a sample from a log-uniform distribution defined over (min, max).
perturb_probability: The probability of actually perturbing the hyperparameter value
when calling perturb(). Must be a float value with 0 <= value <= 1..
perturb_max_pct_change: Maximum allowed change of the hyperparameter value in percent,
when calling perturb() and if perturb() actually perturbs the value (see perturb_probability). Resulting change lies in the range of (-perturb_max_pct_change, perturb_max_pct_change). Must be a float value > 0.
dtype: tf.dtype used by the tf.Variable that holds the hyperparameter value.
Defaults to tf.float32.

name: Name for this FloatHyperparameter instance. Defaults to ‘float_hyperparameter’.

get()

Returns the value of the hyperparameter variable.

perturb(verbose=False)

Perturbs/mutates the value of the hyperparameter variable with perturb_probability.

set(value: float)

Sets the value of the hyperparameter variable.

Args:
value: Value of type float or numpy float.
variable

Returns the tf.Variable that holds the hyperparameter value.

Categorical Integer Hyperparameter

class ftw.tf.hyperparameters.ftw_hp.IntHyperparameter(min_value: int, max_value: int, initial_value: Optional[int] = None, perturb_probability: float = 1.0, dtype=tf.float32, name: str = 'int_hyperparameter')

Bases: ftw.tf.hyperparameters.base.Hyperparameter

Hyperparameter with an int value.

Supported initialization and perturbation behaviour of hyperparameter variable:
  • Supplying min_value and max_value parameters, but no initial_value parameter results in: Initialization by drawing a random sample from a categorical distribution defined over the range [min_value, …, max_value]. Perturbation by drawing from the same distribution.
  • Supplying min_value, max_value and initial_value parameters results in: Initialization with initial_value. Perturbation by drawing a random sample from a categorical distribution defined over the range [min_value, …, max_value].

Contains a tf.Variable that stores the hyperparameter value. Consequently, this module is well suited to be used as a hyperparameter variable inside a tf.function, e.g. in population-based training.

__init__(min_value: int, max_value: int, initial_value: Optional[int] = None, perturb_probability: float = 1.0, dtype=tf.float32, name: str = 'int_hyperparameter')
Args:
min_value: Inclusive lower limit of the range used for random initialization (if no init_value is supplied)
and perturbation sampling. Python int or numpy int expected.
max_value: Inclusive upper limit of the range used for random initialization (if no init_value is supplied)
and perturbation sampling. Python int or numpy int expected.
initial_value: Optional. If given, hyperparameter will be initialized with this value
instead of being randomly initialized from the range defined by (min_value, max_value). Python int or numpy int expected.
perturb_probability: The probability of actually perturbing the hyperparameter value
when calling perturb(). Must be a float value with 0 <= value <= 1.
dtype: tf.dtype used by the tf.Variable that holds the hyperparameter value.
Defaults to tf.float32, even though the hyperparameter is an int, since this is often more compatible with common neural network models.

name: Name for this IntHyperparameter instance. Defaults to ‘int_hyperparameter’.

Raises:
ValueError: If invalid constructor arguments were passed.
get()

Returns the value of the hyperparameter variable.

perturb()

Perturbs/mutates the value of the hyperparameter variable with perturb_probability.

set(value: int)

Sets the value of the hyperparameter variable.

Args:
value: Value of type int or numpy.int.
variable

Returns the tf.Variable that holds the hyperparameter value.