Node storage¶
-
exception
larch.nodestore.
NodeCannotBeModified
(node_id)¶ User called start_modification on node that cannot be modified.
-
exception
larch.nodestore.
NodeExists
(node_id)¶ User tried to put a node that already exists in the store.
-
exception
larch.nodestore.
NodeMissing
(node_store, node_id, error=None, error_msg='')¶ A node cannot be found from a NodeStore.
-
class
larch.nodestore.
NodeStore
(allow_writes, node_size, codec)¶ Abstract base class for storing nodes externally.
The
BTree
class itself does not handle external storage of nodes. Instead, it is given an object that implements the API in this class. An actual implementation might keep nodes in memory, or store them on disk using a filesystem, or a database.Node stores deal with nodes as byte strings: the
codec
encodes them before handing them to the store, and decodes them when it gets them from the store.Each node has an identifier that is unique within the store. The identifier is an integer, and the caller makes the following guarantees about it:
- it is a non-negative integer
- new nodes are assigned the next consecutive one
- it is never re-used
Further, the caller makes the following guarantees about the encoded nodes:
- they have a strict upper size limit
- the tree attempts to fill nodes as close to the limit as possible
The size limit is given to the node store at initialization time. It is accessible via the
node_size
property. Implementations of this API must handle that in some suitable way, preferably by inheriting from this class and calling its initializer.self.max_value_size
gives the maximum size of a value stored in a node.A node store additionally stores some metadata, as key/value pairs, where both key and value is a shortish string. The whole pair must fit into a node, but more than one node can be used for metadata.
-
can_be_modified
(node)¶ Can a node be modified?
-
commit
()¶ Make sure all changes to are committed to the store.
Until this is called, there’s no guarantee that any of the changes since the previous commit are persistent.
-
get_metadata
(key)¶ Return value that corresponds to a key.
-
get_metadata_keys
()¶ Return list of all metadata keys.
-
get_node
(node_id)¶ Return a node from the store.
Raise the
NodeMissing
exception if the node is not in the store (has never been, or has been removed). Raise other errors as suitable.
-
get_refcount
(node_id)¶ Return the reference count for a node.
-
list_nodes
()¶ Return list of ids of all nodes in store.
-
max_index_pairs
()¶ Max number of index pairs in an index node.
-
put_node
(node)¶ Put a new node into the store.
-
remove_metadata
(key)¶ Remove a metadata key, and its corresponding value.
-
remove_node
(node_id)¶ Remove a node from the store.
-
save_metadata
()¶ Save metadata persistently, if applicable.
Not all node stores are persistent, and this method is not relevant to them. However, if the user does not call this method, none of the changes they make will be stored persistently even with a persistent store.
-
save_refcounts
()¶ Save refcounts to disk.
This method only applies to node stores that persist.
-
set_metadata
(key, value)¶ Set a metadata key/value pair.
-
set_refcount
(node_id, refcount)¶ Set the reference count for a node.
-
start_modification
(node)¶ Start modification of a node.
User must call this before modifying a node in place.
If a node cannot be modified,
NodeCannotBeModified
exception will be raised.
-
class
larch.nodestore.
NodeStoreTests
¶ Re-useable tests for
NodeStore
implementations.The
NodeStore
base class can’t be usefully instantiated itself. Instead you are supposed to sub-class it and implement the API in a suitable way for yourself.This class implements a number of tests that the API implementation must pass. The implementation’s own test class should inherit from this class, and
unittest.TestCase
.The test sub-class should define a setUp method that sets the following:
self.ns
to an instance of the API implementation sub-classself.node_size
to the node size
Key size (
self.key_bytes
) is always 3.-
assertEqualNodes
(n1, n2)¶ Assert that two nodes are equal.
Equal means same keys, and same values for keys. Nodes can be either leaf or index ones.
-
key_bytes
= 3¶
-
test_get_freezes_node
()¶
-
test_has_no_node_zero_initially
()¶
-
test_lists_no_nodes_initially
()¶
-
test_lists_node_zero
()¶
-
test_node_not_in_store_can_not_be_modified
()¶
-
test_node_with_refcount_0_can_not_be_modified
()¶
-
test_node_with_refcount_1_can_be_modified
()¶
-
test_node_with_refcount_2_can_not_be_modified
()¶
-
test_put_allows_to_overwrite_a_node
()¶
-
test_put_allows_to_overwrite_a_node_after_upload_queue_push
()¶
-
test_put_freezes_node
()¶
-
test_puts_and_gets_same
()¶
-
test_raises_error_when_getting_unknown_key
()¶
-
test_raises_error_when_removing_unknown_key
()¶
-
test_remove_raises_nodemissing_if_node_does_not_exist
()¶
-
test_removes_metadata
()¶
-
test_removes_node
()¶
-
test_removes_node_from_upload_queue_if_one_exists
()¶
-
test_returns_zero_count_for_unknown_node_id
()¶
-
test_sets_existing_metadata
()¶
-
test_sets_max_value_size
()¶
-
test_sets_metadata
()¶
-
test_sets_node_size
()¶
-
test_sets_refcount
()¶
-
test_sets_several_metadata_keys
()¶
-
test_unfreezes_node_when_modification_starts
()¶
-
test_updates_refcount
()¶
-
exception
larch.nodestore.
NodeTooBig
(node, node_size)¶ User tried to put a node that was too big into the store.