Documentation for pulsar 0.9.2. For development docs, go here.

Asynchronous HTTP client

Pulsar ships with a thread safe, fully featured, HttpClient class for multiple asynchronous HTTP requests.

To get started, one builds a client:

>>> from pulsar.apps import http
>>> client = http.HttpClient()

and than makes requests, in a coroutine:

response = yield http.get('http://www.bbc.co.uk')

or (only in python 3):

response = yield from http.get('http://www.bbc.co.uk')

Making requests

Pulsar HTTP client has no dependencies and an API similar to requests:

from pulsar.apps import http
client = http.HttpClient()
response = yield client.get('https://github.com/timeline.json')

response is a HttpResponse object which contains all the information about the request and the result:

>>> request = response.request
>>> print(request.headers)
Connection: Keep-Alive
User-Agent: pulsar/0.8.2-beta.1
Accept-Encoding: deflate, gzip
Accept: */*
>>> response.status_code
200
>>> print(response.headers)
...

The request attribute of HttpResponse is an instance of HttpRequest.

Authentication

Authentication, either basic or digest, can be added to a client by invoking

In either case the authentication is handled by adding additional headers to your requests.

TLS/SSL

Supported out of the box:

client = HttpClient()
client.get('https://github.com/timeline.json')

you can include certificate file and key too, either to a HttpClient or to a specific request:

client = HttpClient(certkey='public.key')
res1 = client.get('https://github.com/timeline.json')
res2 = client.get('https://github.com/timeline.json',
                  certkey='another.key')

Streaming

This is an event-driven client, therefore streaming support is native.

To stream data received from the client one uses the data_processed event handler. For example:

def new_data(response, **kw):
    if response.status_code == 200:
        data = response.recv_body()
        # do something with this data

response = http.get(..., data_processed=new_data)

The response recv_body() method fetches the parsed body of the response and at the same time it flushes it. Check the proxy server example for an application using the HttpClient streaming capabilities.

WebSocket

The http client support websocket upgrades. First you need to have a websocket handler:

from pulsar.apps import ws

class Echo(ws.WS):

    def on_message(self, websocket, message):
        websocket.write(message)

The websocket response is obtained by:

ws = yield http.get('ws://...', websocket_handler=Echo())

Redirects & Decompression

[TODO]

Synchronous Mode

Can be used in synchronous mode:

client = HttpClient(loop=new_event_loop())

Events

Events control the behaviour of the HttpClient when certain conditions occur. They are useful for handling standard HTTP event such as redirects, websocket upgrades, streaming or anything your application requires.

One time events

There are three one time events associated with an HttpResponse object:

  • pre_request, fired before the request is sent to the server. Callbacks receive the response argument.
  • on_headers, fired when response headers are available. Callbacks receive the response argument.
  • post_request, fired when the response is done. Callbacks receive the response argument.

Adding event handlers can be done at client level:

def myheader_handler(response, exc=None):
    if not exc:
        print('got headers!')

client.bind_event('on_headers', myheader_handler)

or at request level:

response = client.get(..., on_headers=myheader_handler)

By default, the HttpClient has one pre_request callback for handling HTTP tunneling, three on_headers callbacks for handling 100 Continue, websocket upgrade and cookies, and one post_request callback for handling redirects.

Many time events

In addition to the three one time events, the Http client supports two additional events which can occur several times while processing a given response:

  • data_received is fired when new data has been received but not yet parsed
  • data_processed is fired just after the data has been parsed by the HttpResponse. This is the event one should bind to when performing http streaming.

both events support handlers with a signature:

def handler(response, data=None):
    ...

where response is the HttpResponse handling the request and data is the raw data received.

API

The main class here is the HttpClient which is a subclass of AbstractClient.

HTTP Client

class pulsar.apps.http.HttpClient(proxy_info=None, cache=None, headers=None, encode_multipart=True, multipart_boundary=None, keyfile=None, certfile=None, cert_reqs=0, ca_certs=None, cookies=None, store_cookies=True, max_redirects=10, decompress=True, version=None, websocket_handler=None, parser=None, trust_env=True, loop=None, client_version=None, timeout=None, pool_size=10, green=False)[source]

A client for HTTP/HTTPS servers.

It handles pool of asynchronous connections.

Parameters:
headers

Default headers for this HttpClient.

Default: DEFAULT_HTTP_HEADERS.

cookies

Default cookies for this HttpClient.

store_cookies

If True it remebers response cookies and send them back to serves.

Default: True

timeout

Default timeout for the connecting sockets. If 0 it is an asynchronous client.

encode_multipart

Flag indicating if body data is by default encoded using the multipart/form-data or application/x-www-form-urlencoded encoding.

It can be overwritten during a request().

Default: True

proxy_info

Dictionary of proxy servers for this client.

pool_size

The size of a pool of connection for a given host.

connection_pools

Dictionary of connection pools for different hosts

green

If True the requests are processed using the greenio application. When operating in this mode, the HttpClient must be used from a greenlet ather than the main one.

DEFAULT_HTTP_HEADERS

Default headers for this HttpClient

connection_pool

Connection Pool factory

alias of Pool

client_version = 'pulsar/0.9.2'

String for the User-Agent header.

max_redirects = 10

Maximum number of redirects.

It can be overwritten on request().

version = 'HTTP/1.1'

Default HTTP request version for this HttpClient.

It can be overwritten on request().

get(url, **kwargs)[source]

Sends a GET request and returns a HttpResponse object.

Params url:url for the new HttpRequest object.
Parameters:**kwargs – Optional arguments for the request() method.
options(url, **kwargs)[source]

Sends a OPTIONS request and returns a HttpResponse object.

Params url:url for the new HttpRequest object.
Parameters:**kwargs – Optional arguments for the request() method.
head(url, **kwargs)[source]

Sends a HEAD request and returns a HttpResponse object.

Params url:url for the new HttpRequest object.
Parameters:**kwargs – Optional arguments for the request() method.
post(url, **kwargs)[source]

Sends a POST request and returns a HttpResponse object.

Params url:url for the new HttpRequest object.
Parameters:**kwargs – Optional arguments for the request() method.
put(url, **kwargs)[source]

Sends a PUT request and returns a HttpResponse object.

Params url:url for the new HttpRequest object.
Parameters:**kwargs – Optional arguments for the request() method.
patch(url, **kwargs)[source]

Sends a PATCH request and returns a HttpResponse object.

Params url:url for the new HttpRequest object.
Parameters:**kwargs – Optional arguments for the request() method.
delete(url, **kwargs)[source]

Sends a DELETE request and returns a HttpResponse object.

Params url:url for the new HttpRequest object.
Parameters:**kwargs – Optional arguments for the request() method.
request(*args, **kwargs)[source]

Constructs and sends a request to a remote server.

It returns a Future which results in a HttpResponse object.

Parameters:
  • method – request method for the HttpRequest.
  • url – URL for the HttpRequest.
  • response – optional pre-existing HttpResponse which starts a new request (for redirects, digest authentication and so forth).
  • params – optional parameters for the HttpRequest initialisation.
Return type:

a Future

close(async=True, timeout=5)[source]

Close all connections.

Fire the finish one time event once done. Return the Future fired by the finish event.

add_basic_authentication(username, password)[source]

Add a HTTPBasicAuth handler to the pre_requests hook.

add_digest_authentication(username, password)[source]

Add a HTTPDigestAuth handler to the pre_requests hook.

HTTP Request

class pulsar.apps.http.HttpRequest(client, url, method, inp_params, headers=None, data=None, files=None, timeout=None, history=None, charset=None, encode_multipart=True, multipart_boundary=None, source_address=None, allow_redirects=False, max_redirects=10, decompress=True, version=None, wait_continue=False, websocket_handler=None, cookies=None, **ignored)[source]

An HttpClient request for an HTTP resource.

This class has a similar interface to urllib.request.Request.

Parameters:
  • files – optional dictionary of name, file-like-objects.
  • allow_redirects – allow the response to follow redirects.
method

The request method

version

HTTP version for this request, usually HTTP/1.1

history

List of past HttpResponse (collected during redirects).

wait_continue

if True, the HttpRequest includes the Expect: 100-Continue header.

address

(host, port) tuple of the HTTP resource

ssl

Context for TLS connections.

If this is a tunneled request and the tunnel connection is not yet established, it returns None.

proxy

Proxy server for this request.

full_url

Full url of endpoint

data

Body of request

encode()[source]

The bytes representation of this HttpRequest.

Called by HttpResponse when it needs to encode this HttpRequest before sending it to the HTTP resource.

has_header(header_name)[source]

Check header_name is in this request headers.

get_header(header_name, default=None)[source]

Retrieve header_name from this request headers.

remove_header(header_name)[source]

Remove header_name from this request.

HTTP Response

class pulsar.apps.http.HttpResponse(loop=None, one_time_events=None, many_times_events=None)[source]

A ProtocolConsumer for the HTTP client protocol.

Initialised by a call to the HttpClient.request method.

There are two events you can yield in a coroutine:

on_headers

fired once the response headers are received.

on_finished

Fired once the whole request has finished

Public API:

status_code

Numeric status code such as 200, 404 and so forth.

Available once the on_headers has fired.

url

The request full url.

cookies

Dictionary of cookies set by the server or None.

recv_body()[source]

Flush the response body and return it.

get_content()[source]

Retrieve the body without flushing

content_string(charset=None, errors=None)[source]

Decode content as a string.

json(charset=None)[source]

Decode content as a JSON object.

decode_content()[source]

Return the best possible representation of the response body.

raise_for_status()[source]

Raises stored HTTPError or URLError, if occured.

info()[source]

Required by python CookieJar.

Return headers.

OAuth1

class pulsar.apps.http.OAuth1(client_id=None, client=None, **kw)[source]

Add OAuth1 authentication to pulsar HttpClient

OAuth2

class pulsar.apps.http.OAuth2(client_id=None, client=None, **kw)[source]

Add OAuth2 authentication to pulsar HttpClient



Table Of Contents

Previous topic

WebSockets

Next topic

Django Application

This Page