greenhouse.pool – Managed Greenlet Pools¶
- class greenhouse.pool.OneWayPool(func, size=10)¶
a pool whose workers have input only
Parameters: - func (function) – the function the workers should run repeatedly
- size (int) – the number of workers to run
this class can be used as a context manager, in which case start() is called at entry and close() is called on exit from the context.
- close()¶
shut down the pool’s workers
this method sets the closing attribute, and once all queued work has been completed it will set the closed attribute
- join(timeout=None)¶
wait for the pool’s input queue to be cleaned out
Note
this method will block until it has no more pending tasks or, if a timeout is provided, it expires
Parameters: timeout (float or None) – maximum time in seconds to wait
- put(*args, **kwargs)¶
place a new item into the pool to be handled by the workers
all positional and keyword arguments will be passed in as the arguments to the function run by the pool’s workers
- start()¶
start the pool’s workers
- class greenhouse.pool.Pool(*args, **kwargs)¶
Bases: greenhouse.pool.OneWayPool
a pool from which the function’s return values can be retrieved
this class supports use as a context manager just as OneWayPool does, and it also supports iteration, in which case get() results are yielded until the pool has been closed and all results pulled.
- close()¶
shut down the pool’s workers
this method sets the closing attribute, lines up the closed attribute to be set once any queued data has been processed, and raises a PoolClosed() exception in any coroutines still blocked on get().
- get()¶
retrieve a result from the pool
if nothing is already completed when this method is called, it will block until something comes back
if the pool’s function exited via exception, that will come back as a result here as well, but will be re-raised in get().
Note
if there is nothing in the pool’s output queue when this method is called, it will block until something is ready
Returns: a return value from one of the function’s invocations if it exited normally Raises: PoolClosed if the pool was closed before a result could be produced for thie call Raises: any exception that was raised inside the worker function
- join(timeout=None)¶
wait for the pool’s input queue to be cleaned out
Note
this method will block until it has no more pending tasks or, if a timeout is provided, it expires
Parameters: timeout (float or None) – maximum time in seconds to wait
- put(*args, **kwargs)¶
place a new item into the pool to be handled by the workers
all positional and keyword arguments will be passed in as the arguments to the function run by the pool’s workers
- start()¶
start the pool’s workers
- class greenhouse.pool.OrderedPool(func, size=10)¶
Bases: greenhouse.pool.Pool
a pool in which the results are produced in order
Pool can produce results in a different order than that in which they were put if one function invocation blocks for longer than another, but this class enforces that nothing will be produced by get() until the next one in the order in which they were put().
this class supports context manager usage and iteration just as its parent class does.
- close()¶
shut down the pool’s workers
this method sets the closing attribute, lines up the closed attribute to be set once any queued data has been processed, and raises a PoolClosed() exception in any coroutines still blocked on get().
- get()¶
retrieve a result from the pool
if nothing is already completed when this method is called, it will block until something comes back
if the pool’s function exited via exception, that will come back as a result here as well, but will be re-raised in get().
Note
if there is nothing in the pool’s output queue when this method is called, it will block until something is ready
Returns: a return value from one of the function’s invocations if it exited normally Raises: PoolClosed if the pool was closed before a result could be produced for thie call Raises: any exception that was raised inside the worker function
- join(timeout=None)¶
wait for the pool’s input queue to be cleaned out
Note
this method will block until it has no more pending tasks or, if a timeout is provided, it expires
Parameters: timeout (float or None) – maximum time in seconds to wait
- put(*args, **kwargs)¶
place a new item into the pool to be handled by the workers
all positional and keyword arguments will be passed in as the arguments to the function run by the pool’s workers
- start()¶
start the pool’s workers
- greenhouse.pool.map(func, items, pool_size=10)¶
a parallelized work-alike to the built-in map function
this function works by creating an OrderedPool and placing all the arguments in put calls, then yielding items produced by the pool’s get method.
Parameters: - func (function) – the mapper function to use
- items (iterable) – the items to use as the mapper’s arguments
- pool_size (int) – the number of workers for the pool – this amounts to the concurrency with which the map is accomplished (default 10)
Returns: a lazy iterator (like python3’s map or python2’s itertools.imap) over the results of the mapping