Multi-process - roboquant

Background¶

One of the downsides of Python is that by default the runtime is single-threaded. This limits the scalability of Python on more powerful multicore computers.

However by running multiple processes, it is still possible to utilize the cores that are available on your machine (typically at the cost of higher memory usage).

Example¶

This example shows how to perform a walk-forward using the multiprocessing package that comes with Python. Each run is over a specific timeframe and set of parameters for the EMA Crossover strategy.

from multiprocessing import get_context
from itertools import product

import roboquant as rq

# Feed with over 25 years of data
feed = rq.feeds.YahooFeed.us_stocks_10(start_date="2000-01-01")

We now create the function that will be invoked in every process with different paramters.

def walk_forward(params: tuple[rq.Timeframe, tuple[int, int]]) -> str:
    """Perform a run over the provided timeframe and EMA parameters
    The return value is the equity value at the end of the run. In general,
    the return value needs to be serialized to be able to pass it back to the
    main process.
    """
    timeframe, (fast, slow) = params
    strategy = rq.strategies.EMACrossover(fast, slow)
    acc = rq.run(feed, strategy, timeframe=timeframe)
    result = f"{timeframe} EMA({fast:2},{slow:2}) ==> {acc.equity():.0f}"
    return result

Using “fork” ensures that the feed object is not being recreated for each process. The pool is created with default number of processes (equal to the number of CPU cores).

with get_context("fork").Pool() as p:

    # Split overall timeframe into 5 equal non-overlapping timeframes
    timeframe_params = feed.timeframe().split(5)

    # EMACrossover parameters, the fast and slow periods
    ema_params = [(3, 5), (5, 7), (10, 15), (15, 21)]

    # All the combinations of parameters (Cartesian product)
    all_params = list(product(timeframe_params, ema_params))

    assert len(all_params) == len(timeframe_params) * len(ema_params)

    # run the walk-forwards in parallel
    results = p.map(walk_forward, all_params)

    for row in results:
        print(row)

[2000-01-03 05:00:00 ― 2005-05-09 14:24:00> EMA( 3, 5) ==> 1551385@USD
[2000-01-03 05:00:00 ― 2005-05-09 14:24:00> EMA( 5, 7) ==> 1513481@USD
[2000-01-03 05:00:00 ― 2005-05-09 14:24:00> EMA(10,15) ==> 1413730@USD
[2000-01-03 05:00:00 ― 2005-05-09 14:24:00> EMA(15,21) ==> 1313017@USD
[2005-05-09 14:24:00 ― 2010-09-13 23:48:00> EMA( 3, 5) ==> 1308847@USD
[2005-05-09 14:24:00 ― 2010-09-13 23:48:00> EMA( 5, 7) ==> 1576011@USD
[2005-05-09 14:24:00 ― 2010-09-13 23:48:00> EMA(10,15) ==> 1725589@USD
[2005-05-09 14:24:00 ― 2010-09-13 23:48:00> EMA(15,21) ==> 1890949@USD
[2010-09-13 23:48:00 ― 2016-01-19 09:12:00> EMA( 3, 5) ==> 1756395@USD
[2010-09-13 23:48:00 ― 2016-01-19 09:12:00> EMA( 5, 7) ==> 1986181@USD
[2010-09-13 23:48:00 ― 2016-01-19 09:12:00> EMA(10,15) ==> 2223934@USD
[2010-09-13 23:48:00 ― 2016-01-19 09:12:00> EMA(15,21) ==> 1948220@USD
[2016-01-19 09:12:00 ― 2021-05-25 18:36:00> EMA( 3, 5) ==> 2451433@USD
[2016-01-19 09:12:00 ― 2021-05-25 18:36:00> EMA( 5, 7) ==> 2541454@USD
[2016-01-19 09:12:00 ― 2021-05-25 18:36:00> EMA(10,15) ==> 3446246@USD
[2016-01-19 09:12:00 ― 2021-05-25 18:36:00> EMA(15,21) ==> 3004503@USD
[2021-05-25 18:36:00 ― 2026-09-30 04:00:00] EMA( 3, 5) ==> 2162066@USD
[2021-05-25 18:36:00 ― 2026-09-30 04:00:00] EMA( 5, 7) ==> 2016837@USD
[2021-05-25 18:36:00 ― 2026-09-30 04:00:00] EMA(10,15) ==> 2056498@USD
[2021-05-25 18:36:00 ― 2026-09-30 04:00:00] EMA(15,21) ==> 2101155@USD