greenhouse.pool – Managed Greenlet Pools¶

class greenhouse.pool.OneWayPool(func, size=10)¶

a pool whose workers have input only

Parameters:	func (function) – the function the workers should run repeatedly size (int) – the number of workers to run

this class can be used as a context manager, in which case start() is called at entry and close() is called on exit from the context.

close()¶

shut down the pool’s workers

this method sets the closing attribute, and once all queued work has been completed it will set the closed attribute

closed¶

the pool has been closed and all put() items have been handled

closing¶

whether close() has been called

join(timeout=None)¶

wait for the pool’s input queue to be cleaned out

Note

this method will block until it has no more pending tasks or, if a timeout is provided, it expires

Parameters:	timeout (float or None) – maximum time in seconds to wait

put(*args, **kwargs)¶

place a new item into the pool to be handled by the workers

all positional and keyword arguments will be passed in as the arguments to the function run by the pool’s workers

start()¶

start the pool’s workers

class greenhouse.pool.Pool(*args, **kwargs)¶

Bases: greenhouse.pool.OneWayPool

a pool from which the function’s return values can be retrieved

this class supports use as a context manager just as OneWayPool does, and it also supports iteration, in which case get() results are yielded until the pool has been closed and all results pulled.

close()¶

shut down the pool’s workers

this method sets the closing attribute, lines up the closed attribute to be set once any queued data has been processed, and raises a PoolClosed() exception in any coroutines still blocked on get().

closed¶

the pool has been closed, all put() items have been retrieved with :meth`get`.

closing¶

whether close() has been called

get()¶

retrieve a result from the pool

if nothing is already completed when this method is called, it will block until something comes back

if the pool’s function exited via exception, that will come back as a result here as well, but will be re-raised in get().

Note

if there is nothing in the pool’s output queue when this method is called, it will block until something is ready

Returns:	a return value from one of the function’s invocations if it exited normally
Raises :	PoolClosed if the pool was closed before a result could be produced for thie call
Raises :	any exception that was raised inside the worker function

join(timeout=None)¶

wait for the pool’s input queue to be cleaned out

Note

this method will block until it has no more pending tasks or, if a timeout is provided, it expires

Parameters:	timeout (float or None) – maximum time in seconds to wait

put(*args, **kwargs)¶

place a new item into the pool to be handled by the workers

all positional and keyword arguments will be passed in as the arguments to the function run by the pool’s workers

start()¶

start the pool’s workers

class greenhouse.pool.OrderedPool(func, size=10)¶

Bases: greenhouse.pool.Pool

a pool in which the results are produced in order

Pool can produce results in a different order than that in which they were put if one function invocation blocks for longer than another, but this class enforces that nothing will be produced by get() until the next one in the order in which they were put().

this class supports context manager usage and iteration just as its parent class does.

close()¶

shut down the pool’s workers

this method sets the closing attribute, lines up the closed attribute to be set once any queued data has been processed, and raises a PoolClosed() exception in any coroutines still blocked on get().

closed¶

the pool has been closed, all put() items have been retrieved with :meth`get`.

closing¶

whether close() has been called

get()¶

retrieve a result from the pool

if nothing is already completed when this method is called, it will block until something comes back

if the pool’s function exited via exception, that will come back as a result here as well, but will be re-raised in get().

Note

if there is nothing in the pool’s output queue when this method is called, it will block until something is ready

Returns:	a return value from one of the function’s invocations if it exited normally
Raises :	PoolClosed if the pool was closed before a result could be produced for thie call
Raises :	any exception that was raised inside the worker function

join(timeout=None)¶

wait for the pool’s input queue to be cleaned out

Note

this method will block until it has no more pending tasks or, if a timeout is provided, it expires

Parameters:	timeout (float or None) – maximum time in seconds to wait

put(*args, **kwargs)¶

place a new item into the pool to be handled by the workers

all positional and keyword arguments will be passed in as the arguments to the function run by the pool’s workers

start()¶

start the pool’s workers

greenhouse.pool.map(func, items, pool_size=10)¶

a parallelized work-alike to the built-in map function

this function works by creating an OrderedPool and placing all the arguments in put calls, then yielding items produced by the pool’s get method.

Parameters:	func (function) – the mapper function to use items (iterable) – the items to use as the mapper’s arguments pool_size (int) – the number of workers for the pool – this amounts to the concurrency with which the map is accomplished (default 10)
Returns:	a lazy iterator (like python3’s map or python2’s itertools.imap) over the results of the mapping

class greenhouse.pool.PoolClosed¶

Bases: exceptions.RuntimeError

Exception raised to wake any coroutines blocked on get calls when a pool is closed