Documentation for pulsar 0.9.2. For development docs, go here.

Object Data Mapper

Pulsar ships with an ultrafast object data mapper (ODM) for managing and retrieving data asynchronously.

Features

  • Built on top of pulsar data store client api
  • Presents a method of associating user-defined Python classes with data-stores collections/tables
  • An instance of a Model is mapped into an item/row in its corresponding collection/table
  • It allows the use of different stores for different models
  • Straightforward CRUD operations
  • Design to be fast, lightweight and non-intrusive

Getting Started

The first step is to create the two Model we are going to use throughout this tutorial:

from datetime import datetime, timedelta

from pulsar.apps.data import odm

class User(odm.Model):
    username = odm.CharField(index=True)
    password = odm.CharField()
    email = odm.CharField(index=True, required=False)
    is_active = odm.BoolField(default=True)


class Session(odm.Model):
    expiry = odm.DateTimeField(
        default=lambda: datetime.now() + timedelta(days=7))
    user = odm.ForeignKey(User)
    data = odm.JSONField()

The first model contains some basic information for a user, while the second represents a session (think of it as a web session for example).

Mapper & Managers

A Model, such as the User class defined above, has no information regarding database, it is purely a dictionary with additional information about fields.

Registration

Registration consists in associating a Model to a Manager via a Mapper. In this way one can have a group of models associated with their managers pointing at their, possibly different, back-end servers. Registration is straightforward and it is achieved by:

from pulsar.apps.data import odm

models = odm.Mapper('redis://127.0.0.1:6379/7')

models.register(User, Session)

The connection string passed as first argument when initialising a Mapper, is the default data store of that Mapper. It is possible to register models to a different data-stores by passing a store parameter to the Mapper.register() method:

models.register(User, store='redis://127.0.0.1:6379/8')

Accessing managers

Given a models Mapper there are two ways one can access a model Manager to perform database queries.

  • Dictionary interface is the most straightforward and intuitive way:

    # Create a Query for Instrument
    query = models[User].query()
    #
    # Create a new Instrument and save it to the backend server
    inst = models[User].create(...)
    
  • Dotted notation is an alternative and more pythonic way of achieving the same manager via an attribute of the Mapper, the attribute name is given by the Model metaclass name. It is, by default, the class name of the model in lower case:

    query = models.user.query()
    user = models.user.create(...)
    

The dotted notation is less verbose than the dictionary notation and, importantly, it allows to access models and their managers without the need to directly import the model definition. All you need is the Mapper. In other words it makes your application less dependent on the actual implementation of a Model.

Create an instance

When creating a new instance of model the callable method its registered Manager should be used:

pippo = models.user(username='pippo', email='pippo@bla.com')

pippo is a instance not yet persistent in the data store.

Multiple backends

The Mapper allows to use your models in several different back-ends without changing the way you query your data. In addition it allows to specify different back-ends for write operations and for read only operations.

To specify a different back-end for read operations one registers a model in the following way:

models.register(User,
                store='redis://127.0.0.1:6379/8,
                read_store='redis://127.0.0.1:6380/1')

Custom managers

When a Mapper registers a Model, it creates a new instance of a Manager and add it to the dictionary of managers. It is possible to supply a custom manager class by specifying the manager_class attribute on the Model:

from pulsar.apps.data import odm

class CustomManager(odm.Manager):

    def special_query(self, ...):
        return self.query().filter(...)


class MyModel(odm.Model):
    ...

    manager_class = CustomManager

Querying Data

Relationships

There are two Field which represent relationships between Model.

One-to-many

The Session model in our example, contains one ForeignKey field which represents a relationship between the Session model and the User model.

In the context of relational databases a foreign key is a referential constraint between two tables. The same definition applies to pulsar odm. The field store the id of a related Model instance.

Key Properties

  • Behind the scenes, stdnet appends _id to the field name to create its field name in the back-end data-server. In other words, the Session model is mapped into a data-store object with the following entries:

    {'id': ...,
     'user_id': ...,
     'expiry': ...
     'data': ...}
    
  • The attribute of a ForeignKey can be used to access the related object. Using the router we created during registration we get a position instance:

    p = router.position.get(id=1)
    p.instrument    # an instance of Instrument
    

    The second statement is equivalent to:

    router.instrument.query().get(id=p.instrument_id)
    

    Note

    The loading of the related object is done, once only, the first time the attribute is accessed. This means, the first time you access a related field on a model instance, there will be a roundtrip to the backend server.

    Behind the scenes, this functionality is implemented by Python descriptors_. This shouldn’t really matter to you, but we point it out here for the curious.

  • Depending on your application, sometimes it makes a lot of sense to use the load_related query method to boost performance when accessing many related fields.

  • When the object referenced by a ForeignKey is deleted, stdnet also deletes the object containing the ForeignKey unless the Field.required attribute of the ForeignKey field is set to False.

Many-to-many

The ManyToManyField can be used to create relationships between multiple elements of two models. It requires a positional argument, the class to which the model is related.

Behind the scenes, stdnet creates an intermediary model to represent the many-to-many relationship. We refer to this as the through model.

Let’s consider the following example:

class Group(odm.StdModel):
    name = odm.SymbolField(unique=True)

class User(odm.StdModel):
    name = odm.SymbolField(unique=True)
    groups = odm.ManyToManyField(Group, related_name='users')

Both the User class and instances of if have the groups attribute which is an instance of A many-to-may stdnet.odm.related.One2ManyRelatedManager. Accessing the manager via the model class or an instance has different outcomes.

The through model

Custom through model

In most cases, the standard through model implemented by stdnet is all you need. However, sometimes you may need to associate data with the relationship between two models.

For these situations, stdnet allows you to specify the model that will be used to govern the many-to-many relationship and pass it to the ManyToManyField constructor via the through argument. Consider this simple example:

from stdnet import odm

class Element(odm.StdModel):
    name = odm.SymbolField()

class CompositeElement(odm.StdModel):
    weight = odm.FloatField()

class Composite(odm.StdModel):
    name = odm.SymbolField()
    elements = odm.ManyToManyField(Element, through=CompositeElement,
                                   related_name='composites')

API

Model

class pulsar.apps.data.odm.model.Model(*args, **kwargs)[source]

A model is a python dict which represents and item/row in a data-store collection/table.

Fields values can be accessed via the dictionary interface:

model['field1']

or the dotted interface:

model.field1

which is equivalent to:

model.get('field1')
_meta

Class attribute which represents the ModelMeta for this model.

set(field, value, modify=True)[source]

Set the value at field

If modify is True, this method is equivalent to:

model[field] = value
get_raw(field, default=None)[source]

Get the raw value at field

This function does not apply field conversion.

to_json()[source]

Return a JSON serialisable dictionary representation.

save()[source]

Commit changes to backend data store.

delete()[source]

Delete this model from backend data store

Model Meta

class pulsar.apps.data.odm.model.ModelMeta(model, fields, app_label=None, table_name=None, name=None, register=True, pkname=None, abstract=False, **kwargs)[source]

A class for storing meta data for a Model class. To override default behaviour you can specify the Meta class as an inner class of Model in the following way:

from pulsar.apps.data import odm

class MyModel(odm.Model):
    timestamp = odm.FloatField()
    ...

    class Meta:
        name = 'custom'
Parameters:
  • register – if True (default), this ModelMeta is registered in the global models hashtable.
  • abstract – Check the abstract attribute.
  • app_label – Check the app_label attribute.
  • name – Check the name attribute.
  • table_name – Check the table_name attribute

This is the list of attributes and methods available. All attributes, but the ones mentioned above, are initialised by the object relational mapper.

abstract

If True, This is an abstract Meta class.

model

Model for which this instance is the database metadata container.

name

Usually it is the Model class name in lower-case, but it can be customised.

app_label

Unless specified it is the name of the directory or file (if at top level) containing the Model definition. It can be customised.

table_name

The table_name which is by default given by <app_label>_<name>.

dfields

dictionary of all Field in model

scalarfields

Ordered list of all Field which are not StructureField. The order is the same as in the Model definition.

indexes

List of Field which are indexes (index attribute set to True).

pk

The Field representing the primary key.

related

Dictionary of RelatedManager for the model. It is created at runtime by the object data mapper.

manytomany

List of ManyToManyField names for the model. This information is useful during registration.

pkname()[source]

Primary key name. A shortcut for self.pk.name.

pk_to_python(value, backend)[source]

Convert the primary key value to a valid python representation.

store_data(instance, store, action)[source]

Generator of field, value pair for the data store.

Perform validation for instance and can raise FieldError if invalid values are stored in instance.

Manager

class pulsar.apps.data.odm.manager.Manager(model, store=None, read_store=None, mapper=None)[source]

Used by the Mapper to link a data Store collection with a Model.

For example:

from pulsar.apps.data import odm

class MyModel(odm.Model):
    group = odm.SymbolField()
    flag = odm.BooleanField()

models = odm.Mapper()
models.register(MyModel)

manager = models[MyModel]

A Model can specify a custom manager by creating a Manager subclass with additional methods:

class MyModelManager(odm.Manager):

    def special_query(self, **kwargs):
        ...

At this point we need to tell the model about the custom manager, and we do so by setting the manager_class class attribute in the Model:

class MyModel(odm.Model):
    ...

    manager_class = MyModelManager
_model

The Model associated with this manager

_store

The Store associated with this manager

_read_store

The Store associated with this manager for read-only operations

_mapper

The Mapper where this Manager is registered

query_class

alias of Query

create_table(*args, **kwargs)[source]

Create the table/collection for the _model

drop_table()[source]

Drop the table/collection for the _model

query()[source]

Build a Query object

get(*args, **kwargs)[source]

Get a single model

filter(**kwargs)[source]

Build a Query object with filtering clauses

begin()[source]

Begin a new Transaction.

create(*args, **kwargs)[source]

Create a new instance of _model and commit to server.

new(*args, **kwargs)

Create a new instance of _model and commit to server.

insert(*args, **kwargs)

Create a new instance of _model and commit to server.

update(instance, **kwargs)[source]

Update an existing instance of _model.

The instance must have already contain the primary key.

save(instance)[source]

Save an instance of _model.

delete(instance)[source]

Delete an existing instance of _model

Mapper

class pulsar.apps.data.odm.mapper.Mapper(default_store, **kw)[source]

A mapper is a mapping of Model to a Manager.

The Manager are registered with a Store:

from asyncstore import odm

models = odm.Mapper(store)
models.register(MyModel, ...)

# dictionary Notation
query = models[MyModel].query()

# or dotted notation (lowercase)
query = models.mymodel.query()

The models instance in the above snippet can be set globally if one wishes to do so.

A Mapper has four events:

  • pre_commit: fired before instances are committed:

    models.bind_event('pre_commit', callback)
    
  • pre_delete: fired before instances are deleted:

    models.bind_event('pre_delete', callback)
    
  • pre_commit: fired after instances are committed:

    models.bind_event('post_commit', callback)
    
  • post_delete: fired after instances are deleted:

    models.bind_event('post_delete', callback)
    
exception ModelNotFound

Raised when a Manager.get() method does not find any model

Mapper.default_store

The default Store for this Mapper.

Used when calling the register() method without explicitly passing a Store.

Mapper.registered_models

List of registered Model.

Mapper.search_engine

The SearchEngine for this Mapper.

This must be created by users and intalled on a mapper via the set_search_engine() method. Check full text search tutorial for information.

Mapper.begin()[source]

Begin a new Transaction

Mapper.set_search_engine(engine)[source]

Set the SearchEngine for this Mapper.

Check full text search tutorial for information.

Mapper.register(*models, **params)[source]

Register one or several Model with this Mapper.

If a model was already registered it does nothing.

Parameters:
  • models – a list of Model
  • store – a Store or a connection string.
  • read_store – Optional Store for read operations. This is useful when the server has a master/slave configuration, where the master accept write and read operations and the slave read only operations (Redis).
  • include_relatedTrue if related models to model needs to be registered. Default True.
  • params – Additional parameters for the create_store() function.
Returns:

a list models registered or a single model if there was only one

Mapper.from_uuid(uuid, session=None)[source]

Retrieve a Model from its universally unique identifier uuid.

If the uuid does not match any instance an exception will raise.

Mapper.flush(exclude=None, include=None, dryrun=False)[source]

Flush registered_models.

Parameters:
  • exclude – optional list of model names to exclude.
  • include – optional list of model names to include.
  • dryrun – Doesn’t remove anything, simply collect managers to flush.
Returns:

Mapper.unregister(model=None)[source]

Unregister a model if provided, otherwise it unregister all registered models.

Return a list of unregistered model managers or None if no managers were removed.

Mapper.register_applications(applications, models=None, stores=None)[source]

A higher level registration method for group of models located on application modules.

It uses the model_iterator() method to iterate through all Model available in applications and register() them.

Parameters:
  • applications – A String or a list of strings representing python dotted paths where models are implemented. Can also be a module or a list of modules.
  • models – Optional list of models to include. If not provided all models found in applications will be included.
  • stores – optional dictionary which map a model or an application to a store connection string.
Return type:

A list of registered Model.

For example:

mapper.register_applications('mylib.myapp')
mapper.register_applications(['mylib.myapp', 'another.path'])
mapper.register_applications(pythonmodule)
mapper.register_applications(['mylib.myapp', pythonmodule])
Mapper.create_tables(*args, **kwargs)[source]

Loop though registered_models and issue the Manager.create_table() method.

Mapper.drop_tables(*args, **kwargs)[source]

Loop though registered_models and issue the Manager.drop_table() method.

Mapper.models_from_model(model, include_related=False, exclude=None)[source]

Generator of all model in model.

Parameters:
  • model – a Model
  • include_related – if True al related models to model are included
  • exclude – optional set of models to exclude
Mapper.model_iterator(application, include_related=True, exclude=None)[source]

A generator of Model classes found in application.

Parameters:application – A python dotted path or an iterable over python dotted-paths where models are defined.

Only models defined in these paths are considered.

Transaction

class pulsar.apps.data.odm.transaction.Transaction(mapper, name=None)[source]

Transaction class for pipelining commands to a Store.

A Transaction is usually obtained via the Mapper.begin() method:

t = models.begin()

or using the with context manager:

with models.begin() as t:
    ...
name

Optional Transaction name

mapper

the Mapper which initialised this transaction.

_commands

dictionary of commands for each Store in this transaction.

deleted

Dictionary of list of ids deleted from the backend server after a commit operation. This dictionary is only available once the transaction has finished.

saved

Dictionary of list of ids saved in the backend server after a commit operation. This dictionary is only available once the transaction has finished.

execute(*args, **kw)[source]

Queue a command in the default data store.

This method does not use the object data mapper.

add(model)[source]

Add a model to the transaction.

Parameters:
  • model – a Model instance. It must be registered with the mapper which created this Transaction.
  • action – Optional CRUD action to perform
Returns:

the model.

update(instance_or_query, **kw)[source]

Update an instance or a query

tstore(store)[source]

Returns the TransactionStore for store

commit()[source]

Commit the transaction.

This method can be invoked once only otherwise an InvalidOperation occurs.

Returns:a Future which results in the list of transaction
wait(callback=None)[source]

Waits for the transaction have finished.

Parameters:callback – optional function called back once the transaction has finished. The function receives one parameter only, the transaction.
Returns:a Future

Field

class pulsar.apps.data.odm.fields.Field(unique=False, primary_key=None, required=None, index=None, hidden=False, as_cache=False, **extras)[source]

Base class of all odm Fields.

Each field is specified as a Model class attribute.

index

Some data stores requires to create indexes when performing specific queries.

Default False.

unique

If True, the field must be unique throughout the model. In this case index is also True.

Default False.

primary_key

If True, this field is the primary key for the model.

A primary key field has the following properties:

  • unique is also True
  • There can be only one in a model
  • If not specified a AutoIdField will be added

Default False.

required

If False, the field is allowed to be null.

Default True.

default

Default value for this field. It can be a callable attribute.

Default None.

name

Field name, created by the odm at runtime.

store_name

The name for the field, created by the get_store_name() method at runtime. For most field, its value is the same as the name. It is the field stored in the backend database.

_meta

The ModelMeta holding the field. Created by the odm at runtime.

charset

The charset used for encoding decoding text.

hidden

If True the field will be hidden from search algorithms.

Default False.

python_type

The python type for the Field.

as_cache

If True the field contains data which is considered cache and therefore always reproducible. Field marked as cache, have required always False.

This attribute is used by the Model.fieldvalue_pairs method which returns a dictionary of field names and values.

Default False.

register_with_model(name, model)[source]

Called during the creation of a the StdModel class when Metaclass is initialised. It fills Field.name and Field.model. This is an internal function users should never call.

add_to_fields()[source]

Add this Field to the fields of model.

get_store_name()[source]

Generate the store_name at runtime

get_default()[source]

Default value for this Field

sql_alchemy_column()[source]

Return a valid Column for SqlAlchemy model.

CharField

class pulsar.apps.data.odm.fields.CharField(unique=False, primary_key=None, required=None, index=None, hidden=False, as_cache=False, **extras)[source]

IntegerField

class pulsar.apps.data.odm.fields.IntegerField(unique=False, primary_key=None, required=None, index=None, hidden=False, as_cache=False, **extras)[source]

FloatField

class pulsar.apps.data.odm.fields.FloatField(unique=False, primary_key=None, required=None, index=None, hidden=False, as_cache=False, **extras)[source]

ForeignKey

class pulsar.apps.data.odm.fields.ForeignKey(model, related_name=None, related_manager_class=None, **kwargs)[source]

A Field defining a one-to-many objects relationship.

It requires a positional argument representing the Model to which the model containing this field is related. For example:

class Folder(odm.Model):
    name = odm.CharField()

class File(odm.Model):
    folder = odm.ForeignKey(Folder, related_name='files')

To create a recursive relationship, an object that has a many-to-one relationship with itself use:

odm.ForeignKey('self')

Behind the scenes, the odm appends _id to the field name to create its field name in the back-end data-server. In the above example, the database field for the File model will have a folder_id field.

related_name

Optional name to use for the relation from the related object back to self.

proxy_class

alias of LazyForeignKey

related_manager_class

alias of OneToManyRelatedManager

FieldError

class pulsar.apps.data.odm.fields.FieldError[source]

Query

class pulsar.apps.data.odm.query.Query(manager, store=None)[source]

A query for data in a model store.

A Query is produced in terms of a given Manager, using the query() method.

filter(*args, **kwargs)[source]

Create a new Query with additional clauses.

The clauses corresponds to where or limit in a SQL SELECT statement.

Params kwargs:dictionary of limiting clauses.

For example:

qs = manager.query().filter(group='planet')
exclude(*args, **kwargs)[source]

Create a new Query with additional clauses.

The clauses correspond to EXCEPT in a SQL SELECT statement.

Using an equivalent example to the filter() method:

qs = manager.query()
result1 = qs.exclude(group='planet')
result2 = qs.exclude(group=('planet','stars'))
union(*args, **kwargs)[source]

Create a new Query obtained form unions.

Parameters:queries – positional Query parameters to create an union with.

For example, lets say we want to have the union of two queries obtained from the filter() method:

query = mymanager.query()
qs = query.filter(field1='bla').union(query.filter(field2='foo'))
intersect(*args, **kwargs)[source]

Create a new Query obtained form intersection.

Parameters:queries – positional Query parameters to create an intersection with.

For example, lets say we want to have the intersection of two queries obtained from the filter() method:

query = mymanager.query()
q1 = query.filter(field2='foo')
qs = query.filter(field1='bla').intersect(q1)

It returns a new Query that automatically follows the foreign-key relationship related

count(*args, **kwargs)[source]

Count the number of objects selected by this Query.

This method is efficient since the Query does not receive any data from the server apart from the number of matched elements.

all(*args, **kwargs)[source]

All objects selected by this Query.

delete(*args, **kwargs)[source]

Delete all objects selected by this Query.

Compiled Query

class pulsar.apps.data.odm.query.CompiledQuery(store, query)[source]

A signature class for implementing a Query in a pulsar data Store.

_query

The underlying Query

_store

The Store executing the query

count()[source]

Count the number of elements matching the query.

all()[source]

Fetch all matching elements from the server.

Return a Future

delete()[source]

Delete all matching elements from the server.

Return a Future

models(data)[source]

Build a list of models from a list of dictionaries.

Uses the Store.build_model() method

Parameters:data – list of dictionaries
Returns:a list of models
aggregate(kwargs)[source]

Aggregate lookup parameters.

Search Engine

class pulsar.apps.data.odm.searchengine.SearchEngine(name, host, loop=None, database=None, user=None, password=None, encoding=None, **kw)[source]

Interface class for a full text search engine

set_mapper(mapper)[source]

Set the Mapper for this search engine.

upload(items)[source]

Upload items to the search engine

create_table(manager)[source]

Invoked when a Manager creates the data store tables.

By default it does nothing, some search engine implementation may need to do something here



Table Of Contents

Previous topic

Datastore Clients

Next topic

Available Clients

This Page