Generic API resources¶

graceful provides you with some set of generic resources in order to help you describe how structured is data in your API. All of them expect that some serializer instance is provided as a class level attribute. Serializer will handle describing resource fields and also translation between resource representation and internal object values that you use inside of your application.

RetrieveAPI¶

RetrieveAPI represents single element serialized resource. In ‘content’ section of GET response it will return single object. On OPTIONSrequest it will return additional field named ‘fields’ that describes all serializer fields.

It expects from you to implement .retrieve(self, params, meta, **kwargs) method handler that retrieves single object (e.g. from some storage) that will be later serialized using provided serializer.

retrieve() accepts following arguments:

params (dict): dictionary of parsed parameters accordingly to definitions provided as resource class atributes.
meta (dict): dictionary of meta parameters anything added to this dict will will be later included in response ‘meta’ section. This can already prepopulated by method that calls this handler.
kwargs (dict): dictionary of values retrieved from route url template by falcon. This is suggested way for providing resource identifiers.

Example usage:

db = SomeDBInterface()
api = application = falcon.API()

class FooResource(RetrieveAPI):
    serializer = RawSerializer()

    def retrieve(self, params, meta, foo_id, **kwargs):
        return db.Foo.get(id=foo_id)

# note url template param that will be passed to `FooResource.get_object()`
api.add_route('foo/{foo_id}', FooResource())

RetrieveUpdateAPI¶

RetrieveUpdateAPI extends RetrieveAPI with capability to update objects with new data from resource representation provided in PUT request body.

It expects from you to implement same handlers as for RetrieveAPI and also new .update(self, params, meta, validated, **kwargs) method handler that updates single object (e.g. in some storage). Updated object may or may not be returned in response ‘content’ section (this is optional)

update() accepts following arguments:

params (dict): dictionary of parsed parameters accordingly to definitions provided as resource class atributes.
meta (dict): dictionary of meta parameters anything added to this dict will will be later included in response ‘meta’ section. This can already prepopulated by method that calls this handler.
validated (dict): dictionary of internal object fields values after converting from representation with full validation performed accordingly to definition contained within serializer instance.
kwargs (dict): dictionary of values retrieved from route url template by falcon. This is suggested way for providing resource identifiers.

If update will return any value it should have same form as return value of retrieve() because it will be again translated into representation with serializer.

Example usage:

db = SomeDBInterface()
api = application = falcon.API()

class FooResource(RetrieveUpdateAPI):
    serializer = RawSerializer()

    def retrieve(self, params, meta, foo_id, **kwargs):
        return db.Foo.get(id=foo_id)

    def update(self, params, meta, foo_id, **kwargs):
        return db.Foo.update(id=foo_id)

# note: url template kwarg that will be passed to
#       `FooResource.get_object()`
api.add_route('foo/{foo_id}', FooResource())

RetrieveUpdateDeleteAPI¶

RetrieveUpdateDeleteAPI extends RetrieveUpdateAPI with capability to delete objects using DELETE requests.

It expects from you to implement same handlers as for RetrieveUpdateAPI and also new .delete(self, params, meta, **kwargs) method handler that deletes single object (e.g. in some storage).

delete() accepts following arguments:

params (dict): dictionary of parsed parameters accordingly to definitions provided as resource class atributes.
meta (dict): dictionary of meta parameters anything added to this dict will will be later included in response ‘meta’ section. This can already prepopulated by method that calls this handler.
kwargs (dict): dictionary of values retrieved from route url template by falcon. This is suggested way for providing resource identifiers.

Example usage:

db = SomeDBInterface()
api = application = falcon.API()

class FooResource(RetrieveUpdateAPI):
    serializer = RawSerializer()

    def retrieve(self, params, meta, foo_id, **kwargs):
        return db.Foo.get(id=foo_id)

    def update(self, params, meta, foo_id, **kwargs):
        return db.Foo.update(id=foo_id)

    def delete(self, params, meta, **kwargs):
        db.Foo.delete(id=foo_id)

# note url template param that will be passed to `FooResource.get_object()`
api.add_route('foo/{foo_id}', FooResource())

ListAPI¶

ListAPI represents list of resource instances. In ‘content’ section of GET response it will return list of serialized objects representations. On OPTIONS request it will return additional field named ‘fields’ that describes all serializer fields.

It expects from you to implement .list(self, params, meta, **kwargs) method handler that retrieves list (or any iterable) of objects (e.g. from some storage) that will be later serialized using provided serializer.

list() accepts following arguments:

params (dict): dictionary of parsed parameters accordingly to definitions provided as resource class atributes.
meta (dict): dictionary of meta parameters anything added to this dict will will be later included in response ‘meta’ section. This can already prepopulated by method that calls this handler.
kwargs (dict): dictionary of values retrieved from route url template by falcon. This is suggested way for providing resource identifiers.

Example usage:

db = SomeDBInterface()
api = application = falcon.API()

class FooListResource(ListAPI):
    serializer = RawSerializer()

    def list(self, params, meta, **kwargs):
        return db.Foo.all(id=foo_id)

# note that in most cases there is no need do define
# variables in url template for list type of resources
api.add_route('foo/', FooListResource())

ListCreateAPI¶

ListCreateAPI extends ListAPI with capability to create new objects with data from resource representation provided in POST or PATCH request body.

It expects from you to implement same handlers as for ListAPI and also new .create(self, params, meta, validated, **kwargs) and (optionally) .create_bulk(self, params, meta, validated, **kwargs) method handlers that are able to create single single and multiple objects (e.g. in some storage). Created object may or may not be returned in response ‘content’ section (this is optional)

create() accepts following arguments:

params (dict): dictionary of parsed parameters accordingly to definitions provided as resource class atributes.
meta (dict): dictionary of meta parameters anything added to this dict will will be later included in response ‘meta’ section. This can already prepopulated by method that calls this handler.
validated (dict): a single dictionary of internal object fields values after converting from representation with full validation performed accordingly to definition contained within serializer instance.
kwargs (dict): dictionary of values retrieved from route url template by falcon. This is suggested way for providing resource identifiers.

create_bulk() accepts following arguments:

params (dict): dictionary of parsed parameters accordingly to definitions provided as resource class atributes.
meta (dict): dictionary of meta parameters anything added to this dict will will be later included in response ‘meta’ section. This can already prepopulated by method that calls this handler.
validated (dict): a list of multiple dictionaries of internal objects’ field values after converting from representation with full validation performed accordingly to definition contained within serializer instance.
kwargs (dict): dictionary of values retrieved from route url template by falcon. This is suggested way for providing resource identifiers.

If create() and create_bulk() return any value then it should have same form compatible with the return value of retrieve() because it will be again translated into representation with serializer. Of course create() should return single instance of resource but create_bulk() should return collection of resources.

Note that default implementation of ListCreateAPI.create_bulk() is very simple and may not be suited for every use case. If you want to use it please refer to Guide for creating resources in bulk.

Example usage:

db = SomeDBInterface()
api = application = falcon.API()

class FooListResource(ListCreateAPI):
    serializer = RawSerializer()

    def list(self, params, meta, **kwargs):
        return db.Foo.all(id=foo_id)

    def create(self, params, meta, validated, **kwargs):
        return db.Foo.create(**validated)

# note that in most cases there is no need do define
# variables in url template for list type of resources
api.add_route('foo/', FooListResource())

Paginated generic resources¶

PaginatedListAPI and PaginatedListCreateAPI are versions of ListAPI and ListAPI classes that support simple pagination with following parameters:

page_size: size of a single response page
page: page count

They also will ‘meta’ section with following information on GET requests:

page_size
page
next - url query string for next page (only if meta['is_more'] exists and is True)
prev - url query string for previous page (None if first page)

Paginated variations of generic list resource do not assume anything about your resources so actual pagination must still be implemented inside of list() handlers. Anyway this class allows you to manage params and meta for pagination in consistent way across all of your resources if you only decide to use it:

db = SomeDBInterface()
api = application = falcon.API()

class FooPaginatedResource(PaginatedListAPI):
    serializer = RawSerializer()

    def list(self, params, meta, **kwargs):
        query = db.Foo.all(id=foo_id).offset(
            params['page'] * params['page_size']
        ).limit(
            params['page_size']
        )

        # use meta['has_more'] to find out if there are
        # any pages behind this one
        if db.Foo.count() > (params['page'] + 1) * params['page_size']:
            meta['has_more'] = True

        return query

api.add_route('foo/', FooPaginatedtResource())

Note

If you don’t like anything about this opinionated meta section that paginated generic resources provide, you can always override it with own add_pagination_meta(params, meta) method handler.

Generic resources without serialization¶

If you don’t like how serializers work there are also two very basic generic resources that does not rely on serializers: Resource and ListResource. They can be extended with mixins found in graceful.resources.mixins module and provide the same method handlers like the generic resources that utilize serializers (i.e. list(), retrieve(), update() and so on). Note that they do not perform anything beyond content-type level serialization.

Guide for creating resources in bulk¶

ListCreateAPI ships with default implementation of create_bulk() method that will call the create() method separately for every resource instance retrieved from request payload. The actual code is following:

def create_bulk(self, params, meta, **kwargs):
     validated = kwargs.pop('validated')
     return [self.create(params, meta, validated=item) for item in validated]

This approach to bulk resource creation may not be the most performant one if you save resource instance to your storage on every create() call. The other concern is whether you care about data consistency in your storage and want to ensure the “all or nothing” semantics. With default bulk creation handler it may be hard to enforce such contraints. Anyway, you can easily override this method to suit your own needs.

There are at least three ways you can handle bulk resource creation in graceful:

Completely separate bulk and single resource creation: allow create() and create_bulk() handlers to have their own separate code responsible for saving data in the storage.
Deffered saves: Allow your create() handler to skip saves if specific keyword parameter is set and then do your saves in th create_bulk() handler.
Utilize your storage transactions: Wrap your data processing with per-request transaction to ensure “all or nothing” semantics on database level.

Completely separate bulk and single resource creation¶

This approach is simplest to implement but makes only sense if the process of your resource creation is very simple and heavily relies on serializers to validate and prepare your data before save.

Assume your API allows to create and retrieve simple documents in some simple storage that may even not be a real database. Good example would be an API dealing with Solr search engine:

from pysolr import Solr

from graceful.serializers import BaseSerializer
from graceful.fields import StringField
from graceful.resources.generic import ListCreateAPI

solr = Solr("<solr url>", "<solr port>")


class DocumentSerializer(BaseSerializer):
    text = StringField("Document content")
    author = StringField(
        "Document author",
        # note: Assume that due to legacy reasons this field
        #       is stored under different name in Solr.
        #       graceful is great in dealing with such problems!
        source="autor_name_t"
    )


class DocumentsAPI(ListCreateAPI):
    def list(self, params, meta, **kwargs):
        return solr.search("*:*")

    def create(self, params, meta, validated, **kwargs):
        solr.add([validated])
        # note: return document back so its representation
        #       can be included in response body
        return validated

Solr search engine is especially good example here because it will not handle well multiple single-ducument save requests and the best approach is to batch them. The pysolr module (popular library for integration with solr) allows you to save multiple documents with single Solr.add() call. Actually, it even encourages you to batch documents using single call because it accepts only list as input argument.

Let’s override the default create_bulk() so it will save all the documents it receives as the validated argument without calling create() handler:

class DocumentsAPI(ListCreateAPI):
    def list(self, params, meta, **kwargs):
        return solr.search("*:*")

    def create(self, params, meta, validated, **kwargs):
        solr.add([validated])
        # note: return document back so its representation
        #       can be included in the response body
        return validated

    def create_bulk(self, params, meta, validated, **kwargs):
        solr.add(validated)
        # note: return documents back so their representation
        #       can be included in the response body
        return validated

Note that above technique works best for simple use cases where the validated argument represents complete data that can be easily saved directly to your storage without any further modification.

If you need any additional processing of resources in your custom create() and create_bulk() methods before saving them to your storage, the code can quickly become hard to mantain. Anyway, you can start with this approach and refactor it later into deferred saves pattern as these two are very alike and offer similar advantages.

Deferred saves¶

In previous section we said that having separate code that independently saves single resource and resources in bulk may not be a best approach if you need to make some additional data processing before saves. No matter if you do a non-serializer-based data validation or talk to some other external services, you will need to duplicate this additional processing code in both handlers. With proper approach you can limit the code duplication by extrating your resource processing procedures to additial methods but it will eventually make things unnecessarily complex and will still be hard to maintain.

A little improvement to previous code is to reuse single resource creation handler in your custom create_bulk() implementation but allow the create() handler to skip saving data to storage on the caller’s demand. Thus any per-resource processing will always stay in the create() handler code and the create_bulk() will be responsible only for saving the data in bulk:

class DocumentsAPI(ListCreateAPI):
    def list(self, params, meta, **kwargs):
        return solr.search("*:*")

    def create(self, params, meta, validated, skip_save=False, **kwargs):
        # do some additional processing like adding defaults etc.
        validated['created_at'] = time.time()

        # note: skip_save defaults to False on ordinary POST requests
        #       this means ``create()`` was called in single-resource mode
        if not skip_save:
            solr.add([validated])

        # note: return document back so its representation
        #       can be included in the response body
        return validated

    def create_bulk(self, params, meta, validated, **kwargs):
        validated = kwargs.pop('validated')

        processed = [
            self.create(params, meta, item, skip_save=True)
            for item in validated
        ]
        solr.add(processed)

        return processed

This way you can be sure that anything you add to the create() handler will also affect the resources created in bulk. Additionally your API is more efficient because it can save the data in bulk with single request to your storage backend instead of making multiple requests.

Utilize your storage transactions¶

Sometimes you may not concerned about the performance of multiple small saves but only want to have the “all or nothing” semantics of the bulk creation method. If the integration with your storage backend allows you to enforce transactions on the block of code you can easily use such feature to make sure that all the separate saves done with create() handler will take effect in the “all or nothing” manner. Good use case for such appoach could be working with any RDBMS that allows to use transactions.

Let’s assume you have a per-request session object that wraps the integration with the storage backend and allows you to set savepoints and commit/rollback transactions. Many ORM layers (e.g. SQLAlchemy) offer such kind of object code for such technique may look very simillar for different storage providers:

# note: example sqlachemy integration could work that way
engine = create_engine("...")
Session = sessionmaker(bind=engine)

class MyAPI(ListCreateAPI):
    def on_post(req, resp, **kwargs):
        # inject session object into kwargs so it can be later
        # used by ``create()`` handler to manipulate storage
        # and manage transaction
        session = Session()
        try:
            super().on_post(req, resp, session=session, **kwargs)
        except:
            session.rollback()
            raise
        else:
            session.commit()

    def on_patch(req, resp, **kwargs):
        # inject session object into kwargs so it can be later
        # used by ``create_bulk()`` handler to manipulate storage
        # and manage transaction
        session = Session()
        try:
            super().on_patch(req, resp, session=session, **kwargs)
        except:
            session.rollback()
            raise
        else:
            session.commit()

graceful

Navigation

Related Topics

Support

Generic API resources¶

RetrieveAPI¶

RetrieveUpdateAPI¶

RetrieveUpdateDeleteAPI¶

ListAPI¶

ListCreateAPI¶

Paginated generic resources¶

Generic resources without serialization¶

Guide for creating resources in bulk¶

Completely separate bulk and single resource creation¶

Deferred saves¶

Utilize your storage transactions¶