Reverb Adders - Reverb submodule¶
Sequence Adder¶
-
class
ftw.adders.reverb.SequenceAdder(client: <sphinx.ext.autodoc.importer._MockObject object at 0x7f0c8318d748>, sequence_length: int, period: int, delta_encoded: bool = False, chunk_length: Optional[int] = None, priority_fns: Optional[Mapping[str, Callable[[PriorityFnInput], float]]] = None, pad_end_of_episode: bool = True)¶ Bases:
acme.adders.reverb.base.ReverbAdderAn adder which adds sequences of fixed length.
-
reset()¶ Resets the adder’s buffer.
-
Submodules¶
ftw.adders.reverb.multi_sequence module¶
Sequence adders.
This implements adders which add multiple sequences.
-
class
ftw.adders.reverb.multi_sequence.MultiSequenceAdder(client: <sphinx.ext.autodoc.importer._MockObject object at 0x7f0c5ad10080>, sequence_lengths: Mapping[str, int], periods: Mapping[str, int], delta_encoded: bool = False, chunk_length: Optional[int] = None, priority_fns: Optional[Mapping[str, Callable[[PriorityFnInput], float]]] = None, should_insert_fns: Optional[Mapping[str, Callable[[acme.adders.reverb.base.PriorityFnInput], bool]]] = None, pad_end_of_episode: bool = True)¶ Bases:
acme.adders.reverb.base.ReverbAdderAn adder which adds multiple sequences of fixed lengths.
- This class is based on acme.adders.reverb.SequenceAdder, with the following enhancements:
- supports multiple sequences, for which the sequence lengths and periods may differ for each sequence.
- supports conditional inserting of sequences, via the should_insert_fns argument of the constructor.
Furthermore, this class implements a similar bugfix (related to padding) as ftw.adders.reverb.SequenceAdder.
-
reset()¶ Resets the adder’s buffer.
ftw.adders.reverb.multi_sequence_test module¶
-
class
ftw.adders.reverb.multi_sequence_test.MultiSequenceAdderTest(*args, **kwargs)¶ Bases:
ftw.adders.reverb.test_utils.AdderTestMixin,absl.testing.parameterized.TestCase-
run_test_adder(adder: acme.adders.reverb.base.ReverbAdder, first: dm_env._environment.TimeStep, steps: Sequence[Tuple[Any, dm_env._environment.TimeStep]], expected_items: Mapping[str, Sequence[Any]])¶ Runs a unit test case for the adder.
- Args:
adder: The instance of base.ReverbAdder that is being tested. first: The first dm_env.TimeStep that is used to call
base.ReverbAdder.add_first().- steps: A sequence of (action, timestep) tuples that are passed to
- base.ReverbAdder.add().
- expected_items: The sequence of items that are expected to be created
- by calling the adder’s add_first() method on first and add() on all of the elements in steps.
-
test_adder_multi_sequence_MixedSeqLenPeriodsNoPaddingTest(sequence_lengths={'buffer_one': 3, 'buffer_two': 4}, periods={'buffer_one': 2, 'buffer_two': 1}, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=3.0, discount=1.0, observation=3)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=5.0, discount=1.0, observation=4)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=0.0, discount=1.0, observation=5)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=-7.0, discount=1.0, observation=6)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=0.0, discount=1.0, observation=7)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=-9.0, discount=0.0, observation=8))), expected_sequences={'buffer_one': ([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ())], [(3, 0, 5.0, 1.0, False, ()), (4, 0, 0.0, 1.0, False, ()), (5, 0, -7.0, 1.0, False, ())], [(5, 0, -7.0, 1.0, False, ()), (6, 0, 0.0, 1.0, False, ()), (7, 0, -9.0, 0.0, False, ())]), 'buffer_two': ([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ()), (4, 0, 0.0, 1.0, False, ())], [(2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ()), (4, 0, 0.0, 1.0, False, ()), (5, 0, -7.0, 1.0, False, ())], [(3, 0, 5.0, 1.0, False, ()), (4, 0, 0.0, 1.0, False, ()), (5, 0, -7.0, 1.0, False, ()), (6, 0, 0.0, 1.0, False, ())], [(4, 0, 0.0, 1.0, False, ()), (5, 0, -7.0, 1.0, False, ()), (6, 0, 0.0, 1.0, False, ()), (7, 0, -9.0, 0.0, False, ())], [(5, 0, -7.0, 1.0, False, ()), (6, 0, 0.0, 1.0, False, ()), (7, 0, -9.0, 0.0, False, ()), (8, 0, 0.0, 0.0, False, ())])}, priority_fns={'buffer_one': <function <lambda> at 0x7f0c3a57a378>, 'buffer_two': <function <lambda> at 0x7f0c3a57a400>}, should_insert_fns={'buffer_one': <function <lambda> at 0x7f0c3a57a488>, 'buffer_two': <function <lambda> at 0x7f0c3a57a510>}, pad_end_of_episode=False)¶
-
test_adder_multi_sequence_MixedSeqLenPeriodsTest(sequence_lengths={'buffer_one': 3, 'buffer_two': 4}, periods={'buffer_one': 2, 'buffer_two': 1}, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=3.0, discount=1.0, observation=3)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=5.0, discount=1.0, observation=4)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=0.0, discount=1.0, observation=5)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=-7.0, discount=1.0, observation=6)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=0.0, discount=1.0, observation=7)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=-9.0, discount=0.0, observation=8))), expected_sequences={'buffer_one': ([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ())], [(3, 0, 5.0, 1.0, False, ()), (4, 0, 0.0, 1.0, False, ()), (5, 0, -7.0, 1.0, False, ())], [(5, 0, -7.0, 1.0, False, ()), (6, 0, 0.0, 1.0, False, ()), (7, 0, -9.0, 0.0, False, ())], [(7, 0, -9.0, 0.0, False, ()), (8, 0, 0.0, 0.0, False, ()), (0, 0, 0.0, 0.0, False, ())]), 'buffer_two': ([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ()), (4, 0, 0.0, 1.0, False, ())], [(2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ()), (4, 0, 0.0, 1.0, False, ()), (5, 0, -7.0, 1.0, False, ())], [(3, 0, 5.0, 1.0, False, ()), (4, 0, 0.0, 1.0, False, ()), (5, 0, -7.0, 1.0, False, ()), (6, 0, 0.0, 1.0, False, ())], [(4, 0, 0.0, 1.0, False, ()), (5, 0, -7.0, 1.0, False, ()), (6, 0, 0.0, 1.0, False, ()), (7, 0, -9.0, 0.0, False, ())], [(5, 0, -7.0, 1.0, False, ()), (6, 0, 0.0, 1.0, False, ()), (7, 0, -9.0, 0.0, False, ()), (8, 0, 0.0, 0.0, False, ())])}, priority_fns={'buffer_one': <function <lambda> at 0x7f0c3a57a158>, 'buffer_two': <function <lambda> at 0x7f0c3a57a1e0>}, should_insert_fns={'buffer_one': <function <lambda> at 0x7f0c3a57a268>, 'buffer_two': <function <lambda> at 0x7f0c3a57a2f0>})¶
-
test_adder_multi_sequence_RewardPredictionTest(sequence_lengths={'nonzero_reward_buffer': 3, 'zero_reward_buffer': 3}, periods={'nonzero_reward_buffer': 1, 'zero_reward_buffer': 1}, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=3.0, discount=1.0, observation=3)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=5.0, discount=1.0, observation=4)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=0.0, discount=1.0, observation=5)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=-7.0, discount=1.0, observation=6)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=0.0, discount=1.0, observation=7)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=-9.0, discount=0.0, observation=8))), expected_sequences={'nonzero_reward_buffer': ([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ())], [(3, 0, 5.0, 1.0, False, ()), (4, 0, 0.0, 1.0, False, ()), (5, 0, -7.0, 1.0, False, ())], [(5, 0, -7.0, 1.0, False, ()), (6, 0, 0.0, 1.0, False, ()), (7, 0, -9.0, 0.0, False, ())]), 'zero_reward_buffer': ([(2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ()), (4, 0, 0.0, 1.0, False, ())], [(4, 0, 0.0, 1.0, False, ()), (5, 0, -7.0, 1.0, False, ()), (6, 0, 0.0, 1.0, False, ())], [(6, 0, 0.0, 1.0, False, ()), (7, 0, -9.0, 0.0, False, ()), (8, 0, 0.0, 0.0, False, ())])}, priority_fns={'nonzero_reward_buffer': <function <lambda> at 0x7f0c3a60aea0>, 'zero_reward_buffer': <function <lambda> at 0x7f0c3a60af28>}, should_insert_fns={'nonzero_reward_buffer': <function <lambda> at 0x7f0c3a57a048>, 'zero_reward_buffer': <function <lambda> at 0x7f0c3a57a0d0>})¶
-
test_adder_sequence_SequenceEarlyTerminationNoPadding(sequence_lengths={'priority_table': 4}, periods={'priority_table': 1}, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=3.0, discount=0.0, observation=3))), expected_sequences=(), priority_fns={'priority_table': <function <lambda> at 0x7f0c3a60ad90>}, should_insert_fns={'priority_table': <function <lambda> at 0x7f0c3a60ae18>}, pad_end_of_episode=False)¶
-
test_adder_sequence_SequenceEarlyTerminationPaddingPeriodOne(sequence_lengths={'priority_table': 4}, periods={'priority_table': 1}, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=3.0, discount=0.0, observation=3))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 0.0, False, ()), (3, 0, 0.0, 0.0, False, ()), (0, 0, 0.0, 0.0, False, ())],), priority_fns={'priority_table': <function <lambda> at 0x7f0c3a60ab70>}, should_insert_fns={'priority_table': <function <lambda> at 0x7f0c3a60abf8>})¶
-
test_adder_sequence_SequenceEarlyTerminationPaddingPeriodTwo(sequence_lengths={'priority_table': 4}, periods={'priority_table': 2}, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=3.0, discount=0.0, observation=3))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 0.0, False, ()), (3, 0, 0.0, 0.0, False, ()), (0, 0, 0.0, 0.0, False, ())],), priority_fns={'priority_table': <function <lambda> at 0x7f0c3a60ac80>}, should_insert_fns={'priority_table': <function <lambda> at 0x7f0c3a60ad08>})¶
-
test_adder_sequence_SequenceEarlyTerminationPeriodOne(sequence_lengths={'priority_table': 3}, periods={'priority_table': 1}, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=3.0, discount=0.0, observation=3))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 0.0, False, ()), (3, 0, 0.0, 0.0, False, ())],), priority_fns={'priority_table': <function <lambda> at 0x7f0c3a60a950>}, should_insert_fns={'priority_table': <function <lambda> at 0x7f0c3a60a9d8>})¶
-
test_adder_sequence_SequenceEarlyTerminationPeriodTwo(sequence_lengths={'priority_table': 3}, periods={'priority_table': 2}, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=3.0, discount=0.0, observation=3))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 0.0, False, ()), (3, 0, 0.0, 0.0, False, ())],), priority_fns={'priority_table': <function <lambda> at 0x7f0c3a60aa60>}, should_insert_fns={'priority_table': <function <lambda> at 0x7f0c3a60aae8>})¶
-
test_adder_sequence_SequencePaddingPeriodThree(sequence_lengths={'priority_table': 4}, periods={'priority_table': 3}, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=3.0, discount=1.0, observation=3)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=5.0, discount=1.0, observation=4)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=7.0, discount=1.0, observation=5)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=9.0, discount=1.0, observation=6)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=11.0, discount=1.0, observation=7)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=13.0, discount=0.0, observation=8))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ()), (4, 0, 7.0, 1.0, False, ())], [(4, 0, 7.0, 1.0, False, ()), (5, 0, 9.0, 1.0, False, ()), (6, 0, 11.0, 1.0, False, ()), (7, 0, 13.0, 0.0, False, ())], [(7, 0, 13.0, 0.0, False, ()), (8, 0, 0.0, 0.0, False, ()), (0, 0, 0.0, 0.0, False, ()), (0, 0, 0.0, 0.0, False, ())]), priority_fns={'priority_table': <function <lambda> at 0x7f0c3a60a840>}, should_insert_fns={'priority_table': <function <lambda> at 0x7f0c3a60a8c8>})¶
-
test_adder_sequence_SequencePaddingPeriodTwo(sequence_lengths={'priority_table': 4}, periods={'priority_table': 2}, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=3.0, discount=1.0, observation=3)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=5.0, discount=1.0, observation=4)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=7.0, discount=0.0, observation=5))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ()), (4, 0, 7.0, 0.0, False, ())], [(3, 0, 5.0, 1.0, False, ()), (4, 0, 7.0, 0.0, False, ()), (5, 0, 0.0, 0.0, False, ()), (0, 0, 0.0, 0.0, False, ())]), priority_fns={'priority_table': <function <lambda> at 0x7f0c3a60a730>}, should_insert_fns={'priority_table': <function <lambda> at 0x7f0c3a60a7b8>})¶
-
test_adder_sequence_SequencePeriodOne(sequence_lengths={'priority_table': 3}, periods={'priority_table': 1}, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=3.0, discount=1.0, observation=3)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=5.0, discount=1.0, observation=4)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=7.0, discount=0.0, observation=5))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ())], [(2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ()), (4, 0, 7.0, 0.0, False, ())], [(3, 0, 5.0, 1.0, False, ()), (4, 0, 7.0, 0.0, False, ()), (5, 0, 0.0, 0.0, False, ())]), priority_fns={'priority_table': <function <lambda> at 0x7f0c3a653ea0>}, should_insert_fns={'priority_table': <function <lambda> at 0x7f0c3a60a488>})¶
-
test_adder_sequence_SequencePeriodTwo(sequence_lengths={'priority_table': 3}, periods={'priority_table': 2}, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=3.0, discount=1.0, observation=3)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=5.0, discount=1.0, observation=4)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=7.0, discount=0.0, observation=5))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ())], [(3, 0, 5.0, 1.0, False, ()), (4, 0, 7.0, 0.0, False, ()), (5, 0, 0.0, 0.0, False, ())]), priority_fns={'priority_table': <function <lambda> at 0x7f0c3a60a620>}, should_insert_fns={'priority_table': <function <lambda> at 0x7f0c3a60a6a8>})¶
-
ftw.adders.reverb.rnn_sequence module¶
Sequence adders.
This implements adders which add sequences or partial trajectories.
-
class
ftw.adders.reverb.rnn_sequence.NonOverlappingRNNSequenceAdder(client: <sphinx.ext.autodoc.importer._MockObject object at 0x7f0c3a8f2c18>, sequence_length: int, delta_encoded: bool = False, chunk_length: Optional[int] = None, priority_fns: Optional[Mapping[str, Callable[[PriorityFnInput], float]]] = None, pad_end_of_episode: bool = True)¶ Bases:
acme.adders.reverb.base.ReverbAdderAn adder which adds non-overlapping sequences of fixed length.
This adder is based on acme.adders.reverb.SequenceAdder, with the following modification: Enables adding sequences, where only the first recurrent core_state of the sequence is included. This saves RAM and can be used for any recurrent agent that is trained on unrolled sequences, if no overlapping of sequences is required.
Must be used in combination with ftw.datasets.make_reverb_rnn_sequence_fifo_sampler_dataset().
-
reset()¶ Resets the adder’s buffer.
-
ftw.adders.reverb.rnn_sequence_test module¶
-
class
ftw.adders.reverb.rnn_sequence_test.RNNSequenceAdderTest(*args, **kwargs)¶ Bases:
ftw.adders.reverb.test_utils.AdderTestMixin,absl.testing.parameterized.TestCase-
run_test_adder(adder: acme.adders.reverb.base.ReverbAdder, first: dm_env._environment.TimeStep, steps: Sequence[Tuple[Any, dm_env._environment.TimeStep]], expected_items: Sequence[Any])¶ Runs a unit test case for the adder.
- Args:
adder: The instance of base.ReverbAdder that is being tested. first: The first dm_env.TimeStep that is used to call
base.ReverbAdder.add_first().- steps: A sequence of (action, timestep) tuples that are passed to
- base.ReverbAdder.add().
- expected_items: The sequence of items that are expected to be created
- by calling the adder’s add_first() method on first and add() on all of the elements in steps.
-
test_adder_MinimalTest(sequence_length=5, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2), {'core_state': 99}), (0, TimeStep(step_type=<StepType.MID: 1>, reward=3.0, discount=1.0, observation=3), {'core_state': 1}), (0, TimeStep(step_type=<StepType.MID: 1>, reward=5.0, discount=1.0, observation=4), {'core_state': 2}), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=7.0, discount=0.0, observation=5), {'core_state': 3})), expected_sequences=(Step(observation=array([1, 2, 3, 4, 5]), action=array([0, 0, 0, 0, 0]), reward=array([2., 3., 5., 7., 0.]), discount=array([1., 1., 1., 0., 0.]), start_of_episode=array([ True, False, False, False, False]), extras={'core_state': array([99])}), ))¶
-
ftw.adders.reverb.sequence module¶
Sequence adders.
This implements adders which add sequences or partial trajectories.
This module is almost identical to the original acme/adders/reverb/sequence.py, with one bugfix: In the original module, sequences were never padded because the calculation of how many steps need to be padded did not work as expected: with an increasing step, the result of this calculation in the original would eventually become negative, with the result that padding never happens. This edit of the module fixes this bug and always pads with zeros, if enabled. The Bugfix can be found in lines 107-117.
-
class
ftw.adders.reverb.sequence.SequenceAdder(client: <sphinx.ext.autodoc.importer._MockObject object at 0x7f0c8318d748>, sequence_length: int, period: int, delta_encoded: bool = False, chunk_length: Optional[int] = None, priority_fns: Optional[Mapping[str, Callable[[PriorityFnInput], float]]] = None, pad_end_of_episode: bool = True)¶ Bases:
acme.adders.reverb.base.ReverbAdderAn adder which adds sequences of fixed length.
-
reset()¶ Resets the adder’s buffer.
-
ftw.adders.reverb.sequence_test module¶
Tests for sequence adders.
This script adapts https://github.com/deepmind/acme/blob/master/acme/adders/reverb/sequence_test.py and
- modifies the EarlyTerminationNoPadding testcase to reflect the behaviour resulting from the padding bugfix in SequenceAdder
- adds a new PaddingPeriodThree testcase
-
class
ftw.adders.reverb.sequence_test.SequenceAdderTest(*args, **kwargs)¶ Bases:
ftw.adders.reverb.test_utils.AdderTestMixin,absl.testing.parameterized.TestCase-
test_adder_EarlyTerminationNoPadding(sequence_length=4, period=1, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=3.0, discount=0.0, observation=3))), expected_sequences=(), pad_end_of_episode=False)¶
-
test_adder_EarlyTerminationPaddingPeriodOne(sequence_length=4, period=1, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=3.0, discount=0.0, observation=3))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 0.0, False, ()), (3, 0, 0.0, 0.0, False, ()), (0, 0, 0.0, 0.0, False, ())],))¶
-
test_adder_EarlyTerminationPaddingPeriodTwo(sequence_length=4, period=2, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=3.0, discount=0.0, observation=3))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 0.0, False, ()), (3, 0, 0.0, 0.0, False, ()), (0, 0, 0.0, 0.0, False, ())],))¶
-
test_adder_EarlyTerminationPeriodOne(sequence_length=3, period=1, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=3.0, discount=0.0, observation=3))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 0.0, False, ()), (3, 0, 0.0, 0.0, False, ())],))¶
-
test_adder_EarlyTerminationPeriodTwo(sequence_length=3, period=2, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=3.0, discount=0.0, observation=3))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 0.0, False, ()), (3, 0, 0.0, 0.0, False, ())],))¶
-
test_adder_PaddingPeriodThree(sequence_length=4, period=3, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=3.0, discount=1.0, observation=3)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=5.0, discount=1.0, observation=4)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=7.0, discount=1.0, observation=5)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=9.0, discount=1.0, observation=6)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=11.0, discount=1.0, observation=7)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=13.0, discount=0.0, observation=8))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ()), (4, 0, 7.0, 1.0, False, ())], [(4, 0, 7.0, 1.0, False, ()), (5, 0, 9.0, 1.0, False, ()), (6, 0, 11.0, 1.0, False, ()), (7, 0, 13.0, 0.0, False, ())], [(7, 0, 13.0, 0.0, False, ()), (8, 0, 0.0, 0.0, False, ()), (0, 0, 0.0, 0.0, False, ()), (0, 0, 0.0, 0.0, False, ())]))¶
-
test_adder_PeriodOne(sequence_length=3, period=1, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=3.0, discount=1.0, observation=3)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=5.0, discount=1.0, observation=4)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=7.0, discount=0.0, observation=5))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ())], [(2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ()), (4, 0, 7.0, 0.0, False, ())], [(3, 0, 5.0, 1.0, False, ()), (4, 0, 7.0, 0.0, False, ()), (5, 0, 0.0, 0.0, False, ())]))¶
-
test_adder_PeriodTwo(sequence_length=3, period=2, first=TimeStep(step_type=<StepType.FIRST: 0>, reward=None, discount=None, observation=1), steps=((0, TimeStep(step_type=<StepType.MID: 1>, reward=2.0, discount=1.0, observation=2)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=3.0, discount=1.0, observation=3)), (0, TimeStep(step_type=<StepType.MID: 1>, reward=5.0, discount=1.0, observation=4)), (0, TimeStep(step_type=<StepType.LAST: 2>, reward=7.0, discount=0.0, observation=5))), expected_sequences=([(1, 0, 2.0, 1.0, True, ()), (2, 0, 3.0, 1.0, False, ()), (3, 0, 5.0, 1.0, False, ())], [(3, 0, 5.0, 1.0, False, ()), (4, 0, 7.0, 0.0, False, ()), (5, 0, 0.0, 0.0, False, ())]))¶
-
ftw.adders.reverb.test_utils module¶
Utilities for testing Reverb adders.
This module is almost identical to acme/adders/reverb/test_utils.py, with the exception of one bugfix. This bugfix ensures that the number/length of observed items matches that of the expected items. In acme’s current state, this is not done and thus, some tests pass even though they should fail. The bugfix can be found in lines 186-188 of this script.
-
class
ftw.adders.reverb.test_utils.AdderTestMixin(*args, **kwargs)¶ Bases:
absl.testing.absltest.TestCaseA helper mixin for testing Reverb adders.
Note that any test inheriting from this mixin must also inherit from something that provides the Python unittest assert methods.
-
run_test_adder(adder: acme.adders.reverb.base.ReverbAdder, first: dm_env._environment.TimeStep, steps: Sequence[Tuple[Any, dm_env._environment.TimeStep]], expected_items: Sequence[Any])¶ Runs a unit test case for the adder.
- Args:
adder: The instance of base.ReverbAdder that is being tested. first: The first dm_env.TimeStep that is used to call
base.ReverbAdder.add_first().- steps: A sequence of (action, timestep) tuples that are passed to
- base.ReverbAdder.add().
- expected_items: The sequence of items that are expected to be created
- by calling the adder’s add_first() method on first and add() on all of the elements in steps.
-
setUp()¶ Hook method for setting up the test fixture before exercising it.
-
-
class
ftw.adders.reverb.test_utils.FakeClient¶ Bases:
objectFake client for testing.
-
writer(max_sequence_length, delta_encoded=False, chunk_length=None)¶
-
-
class
ftw.adders.reverb.test_utils.FakeWriter(max_sequence_length, delta_encoded=False, chunk_length=None)¶ Bases:
objectFake writer for testing.
-
append(timestep)¶
-
close()¶
-
create_item(table, num_timesteps, priority)¶
-
-
ftw.adders.reverb.test_utils.make_sequence(observations)¶ Create a sequence of timesteps of the form first, [second, …, last].
-
ftw.adders.reverb.test_utils.make_trajectory(observations)¶ Make a simple trajectory from a sequence of observations.
- Arguments:
- observations: a sequence of observations.
- Returns:
- a tuple (first, steps) where first contains the initial dm_env.TimeStep object and steps contains a list of (action, step) tuples. The length of steps is given by episode_length.