Documentation for pulsar 0.9.2. For development docs, go here.
Pulsar ships with an ultrafast object data mapper (ODM) for managing and retrieving data asynchronously.
Model
is mapped into an item/row
in its corresponding collection/tableThe first step is to create the two Model
we are going to use
throughout this tutorial:
from datetime import datetime, timedelta
from pulsar.apps.data import odm
class User(odm.Model):
username = odm.CharField(index=True)
password = odm.CharField()
email = odm.CharField(index=True, required=False)
is_active = odm.BoolField(default=True)
class Session(odm.Model):
expiry = odm.DateTimeField(
default=lambda: datetime.now() + timedelta(days=7))
user = odm.ForeignKey(User)
data = odm.JSONField()
The first model contains some basic information for a user, while the second represents a session (think of it as a web session for example).
A Model
, such as the User
class defined above, has no information
regarding database, it is purely a dictionary with additional information
about fields.
Registration consists in associating a Model
to a Manager
via a Mapper
. In this way one can have a group of models associated
with their managers pointing at their, possibly different, back-end servers.
Registration is straightforward and it is achieved by:
from pulsar.apps.data import odm
models = odm.Mapper('redis://127.0.0.1:6379/7')
models.register(User, Session)
The connection string passed as first argument when
initialising a Mapper
, is the default data store
of that Mapper
.
It is possible to register models to a different data-stores by passing
a store
parameter to the Mapper.register()
method:
models.register(User, store='redis://127.0.0.1:6379/8')
Given a models
Mapper
there are two ways one can access a
model Manager
to perform database queries.
Dictionary interface is the most straightforward and intuitive way:
# Create a Query for Instrument
query = models[User].query()
#
# Create a new Instrument and save it to the backend server
inst = models[User].create(...)
Dotted notation is an alternative and more pythonic way of achieving the
same manager via an attribute of the Mapper
, the attribute
name is given by the Model
metaclass name
.
It is, by default, the class name of the model in lower case:
query = models.user.query()
user = models.user.create(...)
The dotted notation is less verbose than the
dictionary notation and, importantly, it allows to
access models and their managers without the need to directly
import the model definition. All you need is the Mapper
.
In other words it makes your application
less dependent on the actual implementation of a Model
.
When creating a new instance of model the callable method its registered
Manager
should be used:
pippo = models.user(username='pippo', email='pippo@bla.com')
pippo
is a instance not yet persistent in the data store.
The Mapper
allows to use your models in several different back-ends
without changing the way you query your data. In addition it allows to
specify different back-ends for write
operations and for read
only
operations.
To specify a different back-end for read operations one registers a model in the following way:
models.register(User,
store='redis://127.0.0.1:6379/8,
read_store='redis://127.0.0.1:6380/1')
When a Mapper
registers a Model
, it creates a new
instance of a Manager
and add it to the dictionary of managers.
It is possible to supply a custom manager class by specifying the
manager_class
attribute on the Model
:
from pulsar.apps.data import odm
class CustomManager(odm.Manager):
def special_query(self, ...):
return self.query().filter(...)
class MyModel(odm.Model):
...
manager_class = CustomManager
There are two Field
which represent relationships between
Model
.
The Session model in our example,
contains one ForeignKey
field which represents a relationship
between the Session
model and
the User
model.
In the context of relational databases a
foreign key is
a referential constraint between two tables.
The same definition applies to pulsar odm. The field store the id
of a
related Model
instance.
Key Properties
Behind the scenes, stdnet appends _id
to the field name to create its
field name in the back-end data-server. In other words, the
Session model is mapped into a data-store
object with the following entries:
{'id': ...,
'user_id': ...,
'expiry': ...
'data': ...}
The attribute of a ForeignKey
can be used to access the related
object. Using the router we created during registration
we get a position instance:
p = router.position.get(id=1)
p.instrument # an instance of Instrument
The second statement is equivalent to:
router.instrument.query().get(id=p.instrument_id)
Note
The loading of the related object is done, once only, the first time the attribute is accessed. This means, the first time you access a related field on a model instance, there will be a roundtrip to the backend server.
Behind the scenes, this functionality is implemented by Python descriptors_. This shouldn’t really matter to you, but we point it out here for the curious.
Depending on your application, sometimes it makes a lot of sense to use the load_related query method to boost performance when accessing many related fields.
When the object referenced by a ForeignKey
is deleted, stdnet also
deletes the object containing the ForeignKey
unless the
Field.required
attribute of the ForeignKey
field is set
to False
.
The ManyToManyField
can be used to create relationships between
multiple elements of two models. It requires a positional argument, the class
to which the model is related.
Behind the scenes, stdnet creates an intermediary model to represent
the many-to-many relationship. We refer to this as the through model
.
Let’s consider the following example:
class Group(odm.StdModel):
name = odm.SymbolField(unique=True)
class User(odm.StdModel):
name = odm.SymbolField(unique=True)
groups = odm.ManyToManyField(Group, related_name='users')
Both the User
class and instances of if have the groups
attribute which
is an instance of A many-to-may stdnet.odm.related.One2ManyRelatedManager
.
Accessing the manager via the model class or an instance has different outcomes.
In most cases, the standard through model implemented by stdnet is all you need. However, sometimes you may need to associate data with the relationship between two models.
For these situations, stdnet allows you to specify the model that will be used
to govern the many-to-many relationship and pass it to the
ManyToManyField
constructor via the through
argument.
Consider this simple example:
from stdnet import odm
class Element(odm.StdModel):
name = odm.SymbolField()
class CompositeElement(odm.StdModel):
weight = odm.FloatField()
class Composite(odm.StdModel):
name = odm.SymbolField()
elements = odm.ManyToManyField(Element, through=CompositeElement,
related_name='composites')
pulsar.apps.data.odm.model.
Model
(*args, **kwargs)[source]¶A model is a python dict
which represents and item/row
in a data-store collection/table.
Fields values can be accessed via the dictionary interface:
model['field1']
or the dotted interface:
model.field1
which is equivalent to:
model.get('field1')
set
(field, value, modify=True)[source]¶Set the value
at field
If modify
is True
, this method is equivalent to:
model[field] = value
pulsar.apps.data.odm.model.
ModelMeta
(model, fields, app_label=None, table_name=None, name=None, register=True, pkname=None, abstract=False, **kwargs)[source]¶A class for storing meta data for a Model
class.
To override default behaviour you can specify the Meta
class
as an inner class of Model
in the following way:
from pulsar.apps.data import odm
class MyModel(odm.Model):
timestamp = odm.FloatField()
...
class Meta:
name = 'custom'
Parameters: |
|
---|
This is the list of attributes and methods available. All attributes, but the ones mentioned above, are initialised by the object relational mapper.
abstract
¶If True
, This is an abstract Meta class.
app_label
¶Unless specified it is the name of the directory or file
(if at top level) containing the Model
definition.
It can be customised.
table_name
¶The table_name which is by default given by <app_label>_<name>
.
scalarfields
¶Ordered list of all Field
which are not
StructureField
.
The order is the same as in the Model
definition.
Dictionary of RelatedManager
for the model
.
It is created at runtime by the object data mapper.
manytomany
¶List of ManyToManyField
names for the model
. This
information is useful during registration.
pk_to_python
(value, backend)[source]¶Convert the primary key value
to a valid python representation.
store_data
(instance, store, action)[source]¶Generator of field, value
pair for the data store
.
Perform validation for instance
and can raise FieldError
if invalid values are stored in instance
.
pulsar.apps.data.odm.manager.
Manager
(model, store=None, read_store=None, mapper=None)[source]¶Used by the Mapper
to link a data Store
collection
with a Model
.
For example:
from pulsar.apps.data import odm
class MyModel(odm.Model):
group = odm.SymbolField()
flag = odm.BooleanField()
models = odm.Mapper()
models.register(MyModel)
manager = models[MyModel]
A Model
can specify a custom manager by
creating a Manager
subclass with additional methods:
class MyModelManager(odm.Manager):
def special_query(self, **kwargs):
...
At this point we need to tell the model about the custom manager, and we do
so by setting the manager_class
class attribute in the
Model
:
class MyModel(odm.Model):
...
manager_class = MyModelManager
query_class
¶alias of Query
begin
()[source]¶Begin a new Transaction
.
pulsar.apps.data.odm.mapper.
Mapper
(default_store, **kw)[source]¶A mapper is a mapping of Model
to a Manager
.
The Manager
are registered with a Store
:
from asyncstore import odm
models = odm.Mapper(store)
models.register(MyModel, ...)
# dictionary Notation
query = models[MyModel].query()
# or dotted notation (lowercase)
query = models.mymodel.query()
The models
instance in the above snippet can be set globally if
one wishes to do so.
A Mapper
has four events:
pre_commit
: fired before instances are committed:
models.bind_event('pre_commit', callback)
pre_delete
: fired before instances are deleted:
models.bind_event('pre_delete', callback)
pre_commit
: fired after instances are committed:
models.bind_event('post_commit', callback)
post_delete
: fired after instances are deleted:
models.bind_event('post_delete', callback)
ModelNotFound
¶Raised when a Manager.get()
method does not find any model
Mapper.
default_store
¶The default Store
for this Mapper
.
Used when calling the register()
method without explicitly
passing a Store
.
Mapper.
search_engine
¶The SearchEngine
for this Mapper
.
This must be created by users and intalled on a mapper via the
set_search_engine()
method.
Check full text search
tutorial for information.
Mapper.
begin
()[source]¶Begin a new Transaction
Mapper.
set_search_engine
(engine)[source]¶Set the SearchEngine
for this Mapper
.
Check full text search tutorial for information.
Mapper.
register
(*models, **params)[source]¶Register one or several Model
with this Mapper
.
If a model was already registered it does nothing.
Parameters: |
|
---|---|
Returns: | a list models registered or a single model if there was only one |
Mapper.
from_uuid
(uuid, session=None)[source]¶Retrieve a Model
from its universally unique identifier
uuid
.
If the uuid
does not match any instance an exception will raise.
Mapper.
flush
(exclude=None, include=None, dryrun=False)[source]¶Flush registered_models
.
Parameters: |
|
---|---|
Returns: |
Mapper.
unregister
(model=None)[source]¶Unregister a model
if provided, otherwise it unregister all
registered models.
Return a list of unregistered model managers or None
if no managers were removed.
Mapper.
register_applications
(applications, models=None, stores=None)[source]¶A higher level registration method for group of models located on application modules.
It uses the model_iterator()
method to iterate
through all Model
available in applications
and register()
them.
Parameters: |
|
---|---|
Return type: | A list of registered |
For example:
mapper.register_applications('mylib.myapp')
mapper.register_applications(['mylib.myapp', 'another.path'])
mapper.register_applications(pythonmodule)
mapper.register_applications(['mylib.myapp', pythonmodule])
Mapper.
create_tables
(*args, **kwargs)[source]¶Loop though registered_models
and issue the
Manager.create_table()
method.
Mapper.
drop_tables
(*args, **kwargs)[source]¶Loop though registered_models
and issue the
Manager.drop_table()
method.
pulsar.apps.data.odm.transaction.
Transaction
(mapper, name=None)[source]¶Transaction class for pipelining commands to a Store
.
A Transaction
is usually obtained via the Mapper.begin()
method:
t = models.begin()
or using the with
context manager:
with models.begin() as t:
...
name
¶Optional Transaction
name
deleted
¶Dictionary of list of ids deleted from the backend server after a
commit operation. This dictionary is only available once the
transaction has finished
.
saved
¶Dictionary of list of ids saved in the backend server after a commit
operation. This dictionary is only available once the transaction has
finished
.
execute
(*args, **kw)[source]¶Queue a command in the default data store.
This method does not use the object data mapper.
add
(model)[source]¶Add a model
to the transaction.
Parameters: |
|
---|---|
Returns: | the |
commit
()[source]¶Commit the transaction.
This method can be invoked once only otherwise an
InvalidOperation
occurs.
Returns: | a Future which results in the list
of transaction |
---|
pulsar.apps.data.odm.fields.
Field
(unique=False, primary_key=None, required=None, index=None, hidden=False, as_cache=False, **extras)[source]¶Base class of all odm
Fields.
Each field is specified as a Model
class attribute.
index
¶Some data stores requires to create indexes when performing specific queries.
Default False
.
unique
¶If True
, the field must be unique throughout the model.
In this case index
is also True
.
Default False
.
primary_key
¶If True
, this field is the primary key for the model.
A primary key field has the following properties:
unique
is also True
AutoIdField
will be addedDefault False
.
required
¶If False
, the field is allowed to be null.
Default True
.
default
¶Default value for this field. It can be a callable attribute.
Default None
.
name
¶Field name, created by the odm
at runtime.
store_name
¶The name for the field, created by the get_store_name()
method at runtime. For most field, its value is the same as the
name
. It is the field stored in the backend database.
charset
¶The charset used for encoding decoding text.
If True
the field will be hidden from search algorithms.
Default False
.
as_cache
¶If True
the field contains data which is considered cache and
therefore always reproducible. Field marked as cache, have
required
always False
.
This attribute is used by the Model.fieldvalue_pairs
method
which returns a dictionary of field names and values.
Default False
.
register_with_model
(name, model)[source]¶Called during the creation of a the StdModel
class when Metaclass
is initialised. It fills
Field.name
and Field.model
. This is an internal
function users should never call.
get_store_name
()[source]¶Generate the store_name
at runtime
sql_alchemy_column
()[source]¶Return a valid Column for SqlAlchemy model.
pulsar.apps.data.odm.fields.
ForeignKey
(model, related_name=None, related_manager_class=None, **kwargs)[source]¶A Field
defining a one-to-many
objects relationship.
It requires a positional argument representing the Model
to which the model containing this field is related. For example:
class Folder(odm.Model):
name = odm.CharField()
class File(odm.Model):
folder = odm.ForeignKey(Folder, related_name='files')
To create a recursive relationship, an object that has a many-to-one relationship with itself use:
odm.ForeignKey('self')
Behind the scenes, the odm appends _id
to the field
name to create
its field name in the back-end data-server. In the above example,
the database field for the File
model will have a folder_id
field.
Optional name to use for the relation from the related object
back to self
.
proxy_class
¶alias of LazyForeignKey
alias of OneToManyRelatedManager
pulsar.apps.data.odm.query.
Query
(manager, store=None)[source]¶A query for data in a model store.
A Query
is produced in terms of a given Manager
,
using the query()
method.
filter
(*args, **kwargs)[source]¶Create a new Query
with additional clauses.
The clauses corresponds to where
or limit
in a
SQL SELECT
statement.
Params kwargs: | dictionary of limiting clauses. |
---|
For example:
qs = manager.query().filter(group='planet')
exclude
(*args, **kwargs)[source]¶Create a new Query
with additional clauses.
The clauses correspond to EXCEPT
in a SQL SELECT
statement.
Using an equivalent example to the filter()
method:
qs = manager.query()
result1 = qs.exclude(group='planet')
result2 = qs.exclude(group=('planet','stars'))
union
(*args, **kwargs)[source]¶Create a new Query
obtained form unions.
Parameters: | queries – positional Query parameters to create an
union with. |
---|
For example, lets say we want to have the union
of two queries obtained from the filter()
method:
query = mymanager.query()
qs = query.filter(field1='bla').union(query.filter(field2='foo'))
intersect
(*args, **kwargs)[source]¶Create a new Query
obtained form intersection.
Parameters: | queries – positional Query parameters to create an
intersection with. |
---|
For example, lets say we want to have the intersection
of two queries obtained from the filter()
method:
query = mymanager.query()
q1 = query.filter(field2='foo')
qs = query.filter(field1='bla').intersect(q1)
It returns a new Query
that automatically
follows the foreign-key relationship related
pulsar.apps.data.odm.query.
CompiledQuery
(store, query)[source]¶A signature class for implementing a Query
in a
pulsar data Store
.