Algorithmic Trading Archives - relataly.com https://www.relataly.com/category/use-case/algorithmic-trading/ The Business AI Blog Sat, 27 May 2023 10:18:10 +0000 en-US hourly 1 https://wordpress.org/?v=7.0 https://www.relataly.com/wp-content/uploads/2023/04/cropped-AI-cat-Icon-White.png Algorithmic Trading Archives - relataly.com https://www.relataly.com/category/use-case/algorithmic-trading/ 32 32 175977316 Automate Crypto Trading with a Python-Powered Twitter Bot and Gate.io Signals https://www.relataly.com/building-a-twitter-bot-for-trading-signals-using-python/3974/ https://www.relataly.com/building-a-twitter-bot-for-trading-signals-using-python/3974/#comments Wed, 19 May 2021 04:57:00 +0000 https://www.relataly.com/?p=3974 This tutorial develops a Twitter bot in Python that will generate automated trading signals. The bot will pull real-time price data on various cryptocurrencies (Bitcoin, Ethereum, Doge, etc.) from the crypto exchange Gate.io and analyze it using predefined rules. Whenever the bot detects a relevant price change, it automatically posts a tweet via Twitter. Simple ... Read more

The post Automate Crypto Trading with a Python-Powered Twitter Bot and Gate.io Signals appeared first on relataly.com.

]]>

This tutorial develops a Twitter bot in Python that will generate automated trading signals. The bot will pull real-time price data on various cryptocurrencies (Bitcoin, Ethereum, Doge, etc.) from the crypto exchange Gate.io and analyze it using predefined rules. Whenever the bot detects a relevant price change, it automatically posts a tweet via Twitter. Simple Twitter bots can proactively inform their audiences about relevant events in the market. Such an event can be a sharp rise or fall in price or a sudden spike in the trading volume. If we examine data for specific price movements, we can also store these events and use them later to train a predictive model.

More advanced signal bots use predictive models to signal when it is appropriate to enter or exit the market. Or the bot executes the buy- and sell-orders directly itself. A well-defined signaling logic can therefore constitute the first step toward algorithmic trading. But one thing at a time. So in this article, we will begin by developing a simple signal bot.

The rest of this article is structured as follows. First, we take a look at the different code modules of the Twitter bot. After that, we’ll implement the other code modules in Python. Finally, we will integrate the modules and run some tests. We will also quickly introduce the APIs used to build the bot.

trading bot machine learning tutorial gateio trading signals python. Midjourney. relataly.com
Bots can do a lot of cool things but should be used with caution. Image created with Midjourney

Different Modules of the Signal Bot

This section briefly describes the conceptual architecture of the Crypto Twitter bot. Its architecture adheres to a modular design pattern and separates into four loosely coupled modules. Each module has a clear function.

  1. The Data Collection Module retrieves price data from the crypto exchange Gate.io. The module sends requests at regular intervals against the gate.io API. The module adds the data to separate data stores – one for each cryptocurrency. It then forwards the data to the preprocessing module.
  2. The Data Preprocessing Module calculates the statistical indicators, such as moving averages or means, which become the basis for the signaling logic.
  3. The Signaling Module searches for relevant events based on the indicator values provided. If a relevant event is detected, it is reported to the communication module.
  4. The Communication Module connects to the Twitter API. As soon as it is informed about a new event, it tweets about this event on Twitter.

Now that you are familiar with the modules of our Crypto Twitter Bot, we can take a look at its underlying APIs.

Components of the Relataly Crypto Signal Bot

About the APIs Used in this Tutorial

In this tutorial, we will be using two APIs:

  • The Gate.io API to fetch price data.
  • Twitter to post Tweets about Trading Signals

The Gate.io API

Firstly, we will be using the Gate.io API to obtain prices for various cryptocurrencies. Gate.io is one of the smaller crypto exchanges in the crypto-verse. However, it offers a wide range of smaller cryptocurrencies, especially those you cannot trade anywhere else. As of now, the gate.io market endpoint does not require authentication to use its essential functions.

Check out our recent relataly gate.io tutorial to learn how to pull data via the gate.io API in Python.

The Twitter API

The second API that our bot will use is the Twitter API. We will use this API via the Python package Tweepy to post crypto price signals. Check out this article if you are looking for a simple code example of submitting tweets via the Twitter API. If you don’t want to use Twitter, you can disable its use in the code.

Posting tweets via the API requires authentication with a valid developer account. You can apply for a developer account for free on the Twitter developer website. Just be aware that the confirmation can sometimes take several days.

Storing the Twitter API Key

Storing API keys in your code can compromise the security of your application. If the code is made public, for example, by publishing it on a code-sharing website like GitHub, anyone who has access to the code can use the API key to make requests to the API and potentially access sensitive information or cause harm to your account or application. A better practice is to import and access the API key from a separate YAML file, from where you can import it into your project. To store the Twitter API Key, create a YAML file with the name “api_config_twitter.yml” and insert your API key into this file as follows:

api_key: “your api key”

Implementing a Twitter Signal Bot using Python

In this article, we will walk through the process of creating a Twitter bot that automatically tweets updates about cryptocurrency prices. The bot will be designed to pull real-time data on cryptocurrency prices from an external API, and then automatically generate and post tweets on a regular basis. By the end of the article, you will have a fully functional Twitter bot that can keep your followers informed about the latest cryptocurrency prices.

Note: You require a Twitter developer account if you want to use the Twitter functionality. Without an account, you can still print out trading signals to yourself, but you will not be able to post them via the Twitter API.

The code is available on the GitHub repository.

View on GitHub Relataly GitHub Repo

Disclaimer: This article does not constitute financial advice. Stock markets can be very volatile and are generally difficult to predict. Predictive models and other forms of analytics applied in this article only illustrate machine learning use cases.

Python Prerequisites

Before starting the coding part, make sure that you have set up your Python 3 environment and required packages. If you don’t have an environment set up yet, you can follow this tutorial to set up the Anaconda environment.

Also, make sure you install all required packages. In this tutorial, we will be working with the following standard packages: 

In addition, we will use the following two packages:

  • Firstly, the gate.io package (package name gate-API) pulls crypto price data from gate.io.
  • Secondly, we will use the Twitter API library Tweepy to post trading signals via the Twitter API.

You can install packages using console commands:

  • pip install <package name>
  • conda install <package name> (if you are using the anaconda packet manager)

Step #1: Regular Retrieval of Price Data

First, we will define a “prices” class to handle the incoming data flow. The prices class contains a “get_latest_prices” attribute that retrieves price information from gate.io. The function regularly calls the gate.io list_ticker market endpoint.

The list_ticker endpoint returns a list of data fields for cryptocurrency pairs. Examples of price pairs are BTC_USD, BTC_ETH, BTC_ADA, etc. We can limit the response to a single price pair by passing a single pair as a variable in the API call. However, it is not possible to restrict the response to multiple pairs. We either get data for a single pair or all pairs. The response contains a list of the following data fields:

Response returned by the Gate.io API list_tickers operation
Overview of the data fields in the response

The following code maintains a separate dictionary for each cryptocurrency pair. The dictionary contains the name of the cryptocurrency pair and a data frame that includes the price data history. Each time the crypto bot receives a new response from the API, it goes through the response, extracts the price data(Price, Volume, etc.), and appends this data to the Data Frame of the respective cryptocurrency pair. Then the information is passed to the preprocessing module.

import pandas as pd
import numpy as np
import json
import requests
import datetime as dt
import logging
import threading
import time
from __future__ import print_function

import tweepy 
import gate_api
from gate_api.exceptions import ApiException, GateApiException
from twitter_secrets import twitter_secrets as ts # place the twitter_secrets file under <User>/anaconda3/Lib

class Prices:
    """Class that uses the gate api to retrieve currency data."""

    def __init__(self, config):
        self._config = config
        self._logger = logging.getLogger(__name__)
        configuration = gate_api.Configuration(host="https://api.gateio.ws/api/v4")
        api_client = gate_api.ApiClient(configuration)
        self._api_instance = gate_api.SpotApi(api_client)
        self._price_history = {}
        self._cont_update_thread = None
        self._stop_cont_update_thread = None
        self._price_history_lock = threading.Lock()

    def get_price_history(self):
        """Returns a dictionary with the price histories for the currencies."""
        return self._price_history, self._price_history_lock

    def get_latest_prices(self):
        """Gets new price data and adds the values to a DataFrame.

        Returns the DataFrame in a dictionary with the currencies as keys."""
        timestamp = dt.datetime.now()
        try:
            api_response = self._api_instance.list_tickers()
        except GateApiException as e:
            logging.warning(
                "Gate api exception, label: %s, message: %s\n" % (e.label, e.message)
            )
            return {}
        except ApiException as e:
            logging.warning("Exception when calling SpotApi->list_tickers: %s\n" % e)
            return {}
        latest_prices = {}
        for response in api_response:
            currency = response.currency_pair
            if "USDT" not in currency or "BEAR" in currency:
                continue
            value_dict = {
                "base_volume": pd.to_numeric(response.base_volume),
                "change_percentage": pd.to_numeric(response.change_percentage),
                "etf_leverage": pd.to_numeric(response.etf_leverage),
                "etf_net_value": pd.to_numeric(response.etf_net_value),
                "etf_pre_net_value": pd.to_numeric(response.etf_pre_net_value),
                "etf_pre_timestamp": response.etf_pre_timestamp,
                "high_24h": pd.to_numeric(response.high_24h),
                "highest_bid": pd.to_numeric(response.highest_bid),
                "high_bid": pd.to_numeric(response.highest_bid),
                "last": pd.to_numeric(response.last),
                "low_24h": pd.to_numeric(response.low_24h),
                "lowest_ask": pd.to_numeric(response.lowest_ask),
                "quote_volume": pd.to_numeric(response.quote_volume),
                "timestamp": timestamp,
            }
            latest_prices[currency] = pd.DataFrame(value_dict, index=[1])
        return latest_prices

    def start_cont_update(self):
        self._stop_cont_update_thread = threading.Event()
        self._stop_cont_update_thread.clear()
        self._cont_update_thread = threading.Thread(
            target=self._cont_update,
            args=(
                self._stop_cont_update_thread,
                self._price_history_lock,
            ),
        )
        self._cont_update_thread.start()
        self._logger.info("Started continuous price logging")

    def _cont_update(self, stop_event, lock):
        """Continuously adds new prices to the price history."""
        while not stop_event.is_set():
            start_time = time.time()
            lock.acquire()
            for currency, df in self.get_latest_prices().items():
                if currency in self._price_history.keys():
                    self._price_history[currency] = self._price_history[
                        currency
                    ].append(df, ignore_index=True)
                else:
                    self._price_history[currency] = df
            lock.release()
            self._logger.debug("Currency_dfs updated")
            self._wait_before_update(start_time)

    def _wait_before_update(self, start_time):
        elapsed_time = time.time() - start_time
        self._logger.debug(f"Elapsed time: {elapsed_time}")
        if elapsed_time > self._config["price_update_delay"]:
            delay = 0
            self._logger.warning(
                #"It took longer to retrieve the price data than the update_delay!"
            )
        else:
            delay = self._config["price_update_delay"] - elapsed_time
        self._logger.debug(f"Waiting {delay}s until next update")
        time.sleep(delay)

Step #2: Calculate Indicator Values

Next, we will define a few functions that process the regular data inflow from gate.io and calculate indicator values for the different cryptocurrencies.

Absolute price values signal the bot that the price moves up or down. However, our signaling logic will primarily work with thresholds on percentage values. These indicators have a p at the end of the name in the code below.

In addition, we will avoid misleading signals by incorporating moving averages into the signaling logic. Moving averages work on historical data, so we have to hand over the price history when we call the “calc_indicators” function. Furthermore, we take over other indicators from the data frame, including the 24h_low and the 24h_high. These indicators give us additional information about the indicators of the preceding price points. We can use them to build more robust trading signals.

All indicators are calculated separately for each crypto pair, passed to a dictionary, and then passed to the signaling logic. In the next step, we can use these indicator values in our signaling rules.

def calc_indicators(price_history):
    indicators = {}
    indicators_over_all = calc_indicators_over_all(price_history)
    for currency, df in price_history.items():
        if len(df) <= 2:
            logging.getLogger().debug(
                f"Skipped '{currency} when calculating indicators due to a lack of information"
            )
            continue
        volume = df["base_volume"].iloc[-1]
        last_price = df["last"].iloc[-1]
        moving_avg_price = df["last"].mean()
        moving_average_volume = df["base_volume"].mean()
        moving_average_deviation_percent = np.round(
            div(last_price, moving_avg_price) - 1, 2
        )

        price_before = df["last"].iloc[-2]
        price_delta = last_price - price_before
        price_delta_p = div(price_delta, last_price)
        price_delta_before = price_before - df["last"].iloc[-3]
        price_delta_p_before = div((price_before - df["last"].iloc[-3]), price_before)
        low_24h = df["low_24h"].iloc[-1]
        high_24h = df["high_24h"].iloc[-1]
        low_high_diff_p = div(high_24h - low_24h, low_24h)
        change_percentage = df["change_percentage"].iloc[-1]

        indicator_values = {
            "last_price": last_price,
            "price_before": price_before,
            "volume": volume,
            "moving_avg_price": moving_avg_price,
            "moving_average_volume": moving_average_volume,
            "moving_average_deviation_percent": moving_average_deviation_percent,
            "price_delta_p": price_delta_p,
            "price_delta": price_delta,
            "price_delta_before": price_delta_before,
            "price_delta_p_before": price_delta_p_before,
            "high_24h": high_24h,
            "low_24h": low_24h,
            "low_high_diff_p": low_high_diff_p,
            "change_percentage": change_percentage,
        }
        indicator_values.update(indicators_over_all)
        indicators[currency] = indicator_values
    return indicators


def calc_indicators_over_all(price_history):
    avg_change_p = 0
    for currency, df in price_history.items():
        avg_change_p += df["change_percentage"].iloc[-1]
    nr_of_currencies = len(price_history)
    avg_change_p = div(avg_change_p, nr_of_currencies)
    values = {
        "avg_change_p": avg_change_p,
    }
    return values


def div(dividend, divisor, alt_value=0.0):
    return dividend / divisor if divisor != 0 else alt_value

Step #3: Define the Signaling Logic of the Twitter Bot

Our bot will use a signaling logic that differentiates between the following price signals:

  • A simple uptick: Price_delta_p must be higher than the threshold (10%) to trigger.
  • A simple downtick: Price_delta_p must be lower than the threshold (10%) to trigger.
  • The bot does also report on new 24-hour lows and highs
  • Another event on which the bot reports is when an up or down price trend begins to accelerate or slows down.
  • The bot reports that when a price performs a trend reversal (pullback and recovery)

Overview of the different trading signals generated by the signaling logic, twitter bot, algorithmic trading
Overview of the different trading signals generated by the signaling logic

Be aware that the price_delta_p measures the percentage deviation from the previous price point. Thus, the signaling logic that our bot has in place is very dependent on the interval in which the bots request new price data. Shorter time intervals will have a lower chance of triggering because more considerable changes typically occur over a longer time. For more details regarding the signaling logic, please view the code below.

def check_signal(currency, indicators, cs_config):
    ind = indicators[currency]
    signal = ''
    if (ind['moving_avg_price'] > 0
            and ind['last_price'] > 0.0
            and abs(ind['price_delta']) > 0.0
            and abs(ind['price_delta_p']) >= cs_config["delta_threshold_p"]
            and ind['volume'] > 0
    ):
        # up
        if ind['price_delta'] > 0:
            movement_type = 'up +'
            if abs(ind['price_delta_p_before']) > cs_config["delta_threshold_p"]:
                if ind['price_delta_before'] <= 0:
                    movement_type = 'recovery from ' + str(ind['price_before']) + ' to ' + str(ind['last_price'])
                else:
                    if ind['price_delta_p'] * (1-cs_config["delta_threshold_p"]) > ind['price_delta_p_before']:
                        movement_type = 'upward trend accelerates +'
                    elif ind['price_delta_p'] < ind['price_delta_p_before'] * (1-cs_config["delta_threshold_p"]):
                        movement_type = 'upward trend slows down +'
                    elif ind['price_delta_p'] * (1+cs_config["delta_threshold_p"]) >= ind['price_delta_p_before'] >= ind['price_delta_p'] * (1-cs_config["delta_threshold_p"]):
                        movement_type = 'upward trend continues +'
        # down
        elif ind['price_delta'] < 0:
            movement_type = 'down '
            if abs(ind['price_delta_p_before']) > cs_config["delta_threshold_p"]:
                if ind['price_delta_before'] > 0:
                    movement_type = 'pullback from ' + str(ind['price_before']) + ' to ' + str(ind['last_price'])
                else:
                    if ind['price_delta_p'] * (1-cs_config["delta_threshold_p"]) > ind['price_delta_p_before']:
                        movement_type = 'down trend accelerates '
                    elif ind['price_delta_p'] * (1+cs_config["delta_threshold_p"]) >= ind['price_delta_p_before'] >= ind['price_delta_p'] * (1-cs_config["delta_threshold_p"]):
                        movement_type = 'down trend continues '
                    elif ind['price_delta_p'] < ind['price_delta_p_before'] * (1+cs_config["delta_threshold_p"]):
                        movement_type = 'downward trend slows down '

        signal = get_signal_log(movement_type, currency, ind['price_delta_p'], ind['last_price'],
                                ind['moving_avg_price'], ind['volume'], ind['price_delta'], ind['change_percentage'],
                                ind['high_24h'], ind['low_24h'], ind['low_high_diff_p'])

        check_24h_peak(currency, ind['last_price'], ind['low_24h'], ind['high_24h'])

    return signal
    # trade_signal


def check_24h_peak(currency, last_price, low_24h, high_24h):
    if last_price < low_24h:
        print(currency + ' new 24h low $' + str(last_price))
    elif last_price > high_24h:
        print(currency + ' new 24h high $' + str(last_price))


def get_signal_log(movement_type, currency, price_delta_p, last_price, moving_avg_price, volume, price_delta,
                   daily_up_p, high_24h, low_24h, low_high_diff_p):
    signal = f'{currency} {movement_type} ' \
             f'{np.round(price_delta_p * 100, 5)}% ' \
             f'MA:${np.round(moving_avg_price, 6)} ' \
             f'last_price:${np.round(last_price, 6)} ' \
             f'price delta:{np.round(price_delta, 6)} ' \
             f'volume:${np.round(volume, 1)} ' \
             f'daily_change:{np.round(daily_up_p, 2)}% ' \
             f'high_24h:${high_24h} ' \
             f'low_24h:${low_24h} ' \
             f'low_high_diff_p:{np.round(low_high_diff_p * 100, 2)}%'
    return signal

Step #4: Send Tweets via Twitter

Next, we define a simple function that calls the Twitter API and tweets our price signal. Because the Twitter API requires authentication, you must provide the API authentication credentials from a valid Twitter developer account.

It’s best not to store the API credentials directly in code. Still not perfect, but slightly better is to keep the data in a separate python file (for example, called “twitter_secrets”) that you put into your package folder (for example, under /anaconda3/Lib), from where you can import it directly into your code. 

# Twitter Consumer API keys
CONSUMER_KEY    = "api123"
CONSUMER_SECRET = "api123"

# Twitter Access token & access token secret
ACCESS_TOKEN    = "api123"
ACCESS_SECRET   = "api123"

BEARER_TOKEN = "api123"

class TwitterSecrets:
    """Class that holds Twitter Secrets"""

    def __init__(self):
        self.CONSUMER_KEY    = CONSUMER_KEY
        self.CONSUMER_SECRET = CONSUMER_SECRET
        self.ACCESS_TOKEN    = ACCESS_TOKEN
        self.ACCESS_SECRET   = ACCESS_SECRET
        self.BEARER_TOKEN   = BEARER_TOKEN
        
        # Tests if keys are present
        for key, secret in self.__dict__.items():
            assert secret != "", f"Please provide a valid secret for: {key}"

twitter_secrets = TwitterSecrets()

Once you have imported the file, you can then load the API credentials from the file in the following way: 

consumer_key = ts.CONSUMER_KEY
consumer_secret = ts.CONSUMER_SECRET
access_token = ts.ACCESS_TOKEN
access_secret = ts.ACCESS_SECRET

### Print API Auth Data (leave disabled for security reasons)
# print(f'consumer_key: {consumer_key}')
# print(f'consumer_secret: {consumer_secret}')
# print(f'access_token: {access_token}')
# print(f'access_secret: {access_token}')

#authenticating to access the twitter API
auth=tweepy.OAuthHandler(consumer_key,consumer_secret)
auth.set_access_token(access_token,access_secret)
api=tweepy.API(auth)

def send_pricechange_tweet(signal):
    api.update_status(f"{signal} \n {relataly_url}")

Step #5 Starting the Crypto Signal Bot

Finally, we can hit the start button of our crypto signal bot. But before we do this, take a look at some configuration options of the bot.

  • CYCLE_DELAY is the standard interval in seconds in which the bot will call the gate.io API.
  • CURRENCY_PAIR is another API parameter limiting the cryptocurrency pairs to specific currency pairs. The bot will scan the entire market with all currency pairs in the standard setting, including all USDT pairs.
  • TWITTER_ACTIVE defines whether the bot posts signals on Twitter. Be aware that your bot may instantly report any signal on your Twitter account if you enable it.
  • RUNS defines the max number of prices that the bot will retrieve before the bot stops.

Now, let’s test the bot:

RUNS = 50 # the bot will stop after 50 price points
CYCLE_DELAY = 20 # the interval for checking the data and retrieving another price point
EVAL_PRICES_DELAY = 10
CURRENCY_PAIR = "" # the bot will retrieve data for all currency pairs listed on gate.io
PRICES_CONFIG = {"price_update_delay": 20}
TWITTER_ACTIVE = False

CHECK_SIGNAL_CONFIG = {
    "moving_avg_threshold_down_p": 0.10,
    "moving_avg_threshold_up_p": 0.10,
    "delta_threshold_p": 0.07,
    'enable_twitter': TWITTER_ACTIVE,
}

if __name__ == "__main__":
    logging.basicConfig(
        level=logging.INFO, format="\033[02m%(asctime)s %(levelname)s: %(message)s"
    )
    logger = logging.getLogger(__name__)
    prices = Prices(PRICES_CONFIG)
    prices.start_cont_update()
    currency_dfs = {}
    logging.info(f"Crypto bot is starting - please wait")
    logger.info(f"Collecting crypto data from gate.io for {EVAL_PRICES_DELAY}s")
    time.sleep(EVAL_PRICES_DELAY)
    logger.info(f"\n<< Crypto signal bot started :-) >>")
    logger.info(f"<< Checking prices every {CYCLE_DELAY} seconds >>")
    logger.info(f"Now checking for signals - please wait\n")
    for i in range(RUNS):
        price_history, lock = prices.get_price_history()
        lock.acquire()
        indicators = calc_indicators(price_history)
        lock.release()
        for currency in indicators.keys():
            if not indicators[currency]:
                continue
            signal = check_signal(
                currency,
                indicators,
                CHECK_SIGNAL_CONFIG,
            )
            if signal:
                logger.info(signal)
                if CHECK_SIGNAL_CONFIG['enable_twitter']:
                    send_pricechange_tweet(signal)
                    print('send via twitter')
        time.sleep(CYCLE_DELAY)
2022-03-09 11:40:38,939 INFO: Started continuous price logging
2022-03-09 11:40:38,940 INFO: Crypto bot is starting - please wait
2022-03-09 11:40:38,940 INFO: Collecting crypto data from gate.io for 10s
2022-03-09 11:40:48,941 INFO: 
<< Crypto signal bot started :-) >>
2022-03-09 11:40:48,942 INFO: << Checking prices every 20 seconds >>
2022-03-09 11:40:48,942 INFO: Now checking for signals - please wait

2022-03-09 11:52:06,800 INFO: EOSBULL_USDT up + 19.42446% MA:$1.1e-05 last_price:$1.4e-05 price delta:3e-06 volume:$1272326905.1 daily_change:33.65% high_24h:$1.16e-05 low_24h:$9.8e-06low_high_diff_p:18.37%
EOSBULL_USDT new 24h high $1.39e-05
send via twitter

And this is what the tweets will look like on Twitter:

output of our twitter bot, signalling logic, algorithmic trading, crypto price bot, gateio

Summary

Congratulations on completing this tutorial! In this article, you learned how to build a Python-based Twitter crypto signal bot. When run, the bot will regularly retrieve cryptocurrency quotes from the Gate.io exchange and tweet about any price movements based on a simple signaling logic.

While the signaling logic in this tutorial is kept simple, this basic framework provides a foundation for you to further develop and enhance the signaling rules. For example, you could consider using volume or price volatility changes as the basis for defining signals. Have fun experimenting and expanding upon this project!

If you found this article helpful, please show your appreciation by leaving a comment. Cheers

Sources and Further Reading

  1. Charu C. Aggarwal (2018) Neural Networks and Deep Learning
  2. Jansen (2020) Machine Learning for Algorithmic Trading: Predictive models to extract signals from market and alternative data for systematic trading strategies with Python
  3. Aurélien Géron (2019) Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems
  4. David Forsyth (2019) Applied Machine Learning Springer
  5. Andriy Burkov (2020) Machine Learning Engineering

The links above to Amazon are affiliate links. By buying through these links, you support the Relataly.com blog and help to cover the hosting costs. Using the links does not affect the price.

The post Automate Crypto Trading with a Python-Powered Twitter Bot and Gate.io Signals appeared first on relataly.com.

]]> https://www.relataly.com/building-a-twitter-bot-for-trading-signals-using-python/3974/feed/ 1 3974 Stock Market Prediction – Adjusting Time Series Prediction Intervals in Python https://www.relataly.com/changing-prediction-intervals-for-time-series-forecasting-models/169/ https://www.relataly.com/changing-prediction-intervals-for-time-series-forecasting-models/169/#comments Wed, 01 Apr 2020 16:37:56 +0000 http://www.relataly.com/?p=169 Get ready to level up your time-series forecasting game! In this tutorial, we’re going to take things up a notch by showing you how to adjust prediction intervals using Keras recurrent neural networks and Python. Now, you may remember our previous article on stock market forecasting where we made a forecast for the S&P500 stock ... Read more

The post Stock Market Prediction – Adjusting Time Series Prediction Intervals in Python appeared first on relataly.com.

]]>

Get ready to level up your time-series forecasting game! In this tutorial, we’re going to take things up a notch by showing you how to adjust prediction intervals using Keras recurrent neural networks and Python.

Now, you may remember our previous article on stock market forecasting where we made a forecast for the S&P500 stock market index using a prediction interval of just one day. However, other time-series prediction problems may require us to look further ahead – maybe several days, weeks, or even months. Fear not! We’ve got you covered.

By tweaking our data preparation and model architecture, we can modify the prediction interval and create a single-step forecast for a longer time frame. In this article, we’re going to show you exactly how to do that.

First, we’ll give you a quick rundown of different methods for adjusting the time series prediction interval. Then, we’ll dive into the practical stuff. We’ll use Python to train a simple neural network on stock market data and validate its performance. Once we’re happy with the model, we’ll prepare the data in a way that allows us to forecast a single but more extended step into the future.

Ready to take your time-series forecasting to the next level? Then let’s get started!

Disclaimer

This article does not constitute financial advice. Stock markets can be very volatile and are generally difficult to predict. Predictive models and other forms of analytics applied in this article only serve the purpose of illustrating machine learning use cases.

Time Series Analysis Clocks
Time Series Analysis

Ways of Adjusting Prediction Intervals

When forecasting one step, the prediction interval is the point in time for which a prediction model will simulate the next value. There are three different ways to change the prediction interval:

  • Multi-Step Rolling Forecasting: Another way is to train the model on its output. We do this by maintaining the predictions and reusing them as input in the subsequent training run. In this way, the projections range one-time step further ahead with each iteration. After seven iterations, based on daily input time steps, the model will have provided the output for a weekly prediction. We have covered this rolling forecasting approach in a separate tutorial.
  • Deep Multi-Output Forecasting: A third option is to create a multi-output model that provides an entire series of predictions with multiple timesteps. We have covered this multi-output forecasting approach in a separate tutorial.
  • Single-step forecasting with bigger timesteps: In a single-stage forecasting approach, the input data defines the length of a time step. Changing the size of the input steps will change the output steps to the same extent. For example, a model that uses daily prices as input data will also provide day-to-day forecasts. We will cover this forecasting approach in the following section.

Predicting the Price of the S&P500 One Week Ahead

Let’s begin with the hands-on part. In this part, we will use Python to create a single-step forecasting model with more extended timesteps. Our model will make projections that reach one week ahead. For this purpose, we reuse most of the code from the previous article on univariate single-step daily forecasting. So we won’t go into all the details and will only speak about the areas in which we must adjust the code. Changes are necessary to data preparation and model architecture.

In the following, we develop a single-variate neural network model that forecasts the S&P500 stock market index. The code is available on the GitHub repository.

View on GitHub Relataly GitHub Repo

Prerequisites

Before starting the coding part, make sure that you have set up your Python 3 environment and required packages. If you don’t have an environment, you can follow these steps to set up the Anaconda environment.

Also, make sure you install all required packages. In this tutorial, we will be working with the following standard packages: 

In addition, we will be using Keras (2.0 or higher) with Tensorflow backend, the machine learning library sci-kit-learn, and pandas DataReader to interact with the yahoo finance API.

You can install packages using console commands:

  • pip install <package name>
  • conda install <package name> (if you are using the anaconda packet manager)

Step #1 Load the Data

In the following, we will modify the prediction interval of the neural network model we developed in a previous post. As a result, the model will generate predictions for the market price of the S&P500 Index that range one week ahead.

As before, we start loading the stock market data via an API. 

# A tutorial for this file is available at www.relataly.com

import math # Fundamental package for scientific computing with Python
import numpy as np # Additional functions for analysing and manipulating data
import pandas as pd # Date Functions
from datetime import date, timedelta # This function adds plotting functions for calender dates
from pandas.plotting import register_matplotlib_converters # Important package for visualization - we use this to plot the market data
import matplotlib.pyplot as plt # Formatting dates
import matplotlib.dates as mdates # Packages for measuring model performance / errors
from sklearn.metrics import mean_absolute_error, mean_squared_error # Tools for predictive data analysis. We will use the MinMaxScaler to normalize the price data 
from sklearn.preprocessing import MinMaxScaler # Deep learning library, used for neural networks
from tensorflow.keras.models import Sequential # Deep learning classes for recurrent and regular densely-connected layers
from tensorflow.keras.layers import LSTM, Dense
from tensorflow.keras.callbacks import EarlyStopping
import tensorflow as tf
import seaborn as sns
sns.set_style('white', { 'axes.spines.right': False, 'axes.spines.top': False})

# check the tensorflow version and the number of available GPUs
print('Tensorflow Version: ' + tf.__version__)
physical_devices = tf.config.list_physical_devices('GPU')
print("Num GPUs:", len(physical_devices))


# Setting the timeframe for the data extraction
end_date = date.today().strftime("%Y-%m-%d")
start_date = '2010-01-01'

# Getting S&P500 quotes
stockname = 'S&P500'
symbol = '^GSPC'

# You can either use webreader or yfinance to load the data from yahoo finance
# import pandas_datareader as webreader
# df = webreader.DataReader(symbol, start=start_date, end=end_date, data_source="yahoo")

import yfinance as yf #Alternative package if webreader does not work: pip install yfinance
df = yf.download(symbol, start=start_date, end=end_date)

df.head(5)
Tensorflow Version: 2.5.0
Num GPUs: 0
[*********************100%***********************]  1 of 1 completed
			Open		High		Low			Close		Adj Close	Volume
Date						
2009-12-31	1126.599976	1127.640015	1114.810059	1115.099976	1115.099976	2076990000
2010-01-04	1116.560059	1133.869995	1116.560059	1132.989990	1132.989990	3991400000
2010-01-05	1132.660034	1136.630005	1129.660034	1136.520020	1136.520020	2491020000
2010-01-06	1135.709961	1139.189941	1133.949951	1137.140015	1137.140015	4972660000
2010-01-07	1136.270020	1142.459961	1131.319946	1141.689941	1141.689941	5270680000

Step #2 Adjusting the Shape of the Input Data and Exploration

We have a DataFrame that contains the daily price quotes for the S&P 500. Next, we prepare the data to include the weekly price quotes. If we want our model to provide weekly price predictions, we need to change the data so that the input contains weekly price quotes. A simple way to achieve this is to iterate through the rows and only keep every 7th row. 

# Changing the data structure to a dataframe with weekly price quotes
df["index1"] = range(1, len(df) + 1)
rownumber = df.shape[0]
lst = list(range(rownumber))
list_of_relevant_numbers = lst[0::7]
df_weekly = df[df["index1"].isin(list_of_relevant_numbers)]
df_weekly.head(5)
	Open	High	Low	Close	Adj Close	Volume	index1
Date							
2010-01-11	1145.959961	1149.739990	1142.020020	1146.979980	1146.979980	4255780000	7
2010-01-21	1138.680054	1141.579956	1114.839966	1116.479980	1116.479980	6874290000	14
2010-02-01	1073.890015	1089.380005	1073.890015	1089.189941	1089.189941	4077610000	21
2010-02-10	1069.680054	1073.670044	1059.339966	1068.130005	1068.130005	4251450000	28
2010-02-22	1110.000000	1112.290039	1105.380005	1108.010010	1108.010010	3814440000	35

After this, we quickly create a line plot to validate that everything looks as expected.

# Creating a Lineplot
years = mdates.YearLocator() 
fig, ax1 = plt.subplots(figsize=(16, 6))
ax1.xaxis.set_major_locator(years)
ax1.legend([stockname], fontsize=12)
plt.title(stockname + ' from '+ start_date + ' to ' + end_date)
sns.lineplot(data=df['Close'], label=stockname, linewidth=1.0)
plt.ylabel('S&P500 Points')
plt.show()
			Open		High		Low			Close		Adj Close	Volume		index1
Date							
2010-01-11	1145.959961	1149.739990	1142.020020	1146.979980	1146.979980	4255780000	7
2010-01-21	1138.680054	1141.579956	1114.839966	1116.479980	1116.479980	6874290000	14
2010-02-01	1073.890015	1089.380005	1073.890015	1089.189941	1089.189941	4077610000	21
2010-02-10	1069.680054	1073.670044	1059.339966	1068.130005	1068.130005	4251450000	28
2010-02-22	1110.000000	1112.290039	1105.380005	1108.010010	1108.010010	3814440000	35

Step #3 Preprocess the Data

Before we can train the neural network, we first need to define the shape of the training data. We use weekly price quotes and define an input_sequence_length of 50 weeks.

# Feature Selection - Only Close Data
train_df = df.filter(['Close'])
data_unscaled = df.values

# Transform features by scaling each feature to a range between 0 and 1
mmscaler = MinMaxScaler(feature_range=(0, 1))
np_data = mmscaler.fit_transform(data_unscaled)

# Creating a separate scaler that works on a single column for scaling predictions
scaler_pred = MinMaxScaler()
df_Close = pd.DataFrame(df['Close'])
np_Close_scaled = scaler_pred.fit_transform(df_Close)

# Set the sequence length - this is the timeframe used to make a single prediction
sequence_length = 25

# Prediction Index
index_Close = train_df.columns.get_loc("Close")

# Split the training data into train and train data sets
# As a first step, we get the number of rows to train the model on 80% of the data 
train_data_length = math.ceil(np_data.shape[0] * 0.8)

# Create the training and test data
train_data = np_data[0:train_data_length, :]
test_data = np_data[train_data_length - sequence_length:, :]

# The RNN needs data with the format of [samples, time steps, features]
# Here, we create N samples, sequence_length time steps per sample, and 6 features
def partition_dataset(sequence_length, train_df):
    x, y = [], []
    data_len = train_df.shape[0]
    for i in range(sequence_length, data_len):
        x.append(train_df[i-sequence_length:i,:]) #contains sequence_length values 0-sequence_length * columsn
        y.append(train_df[i, index_Close]) #contains the prediction values for validation (3rd column = Close),  for single-step prediction
    
    # Convert the x and y to numpy arrays
    x = np.array(x)
    y = np.array(y)
    return x, y

# Generate training data and test data
x_train, y_train = partition_dataset(sequence_length, train_data)
x_test, y_test = partition_dataset(sequence_length, test_data)

# Print the shapes: the result is: (rows, training_sequence, features) (prediction value, )
print(x_train.shape, y_train.shape)
print(x_test.shape, y_test.shape)

# Validate that the prediction value and the input match up
# The last close price of the second input sample should equal the first prediction value
print(x_test[1][sequence_length-1][index_Close])
print(y_test[0])
(2465, 25, 7) (2465,)
(622, 25, 7) (622,)
0.5509444640253316
0.5509444640253316

Step #4 Building a Time Series Prediction Model

The first layer of neurons in our neural network needs to fit the input values from the data. Therefore, we need 50 neurons – one for each input price quote.

The generic model architecture of the recurrent neural network for time series prediction
The model architecture of the recurrent neural network

We use the following input arguments for the model fit:

  • x_train: Vector, matrix, or array of training data. It can also be a list (as in our case) if the model has multiple inputs.
  • y_train: Vector, matrix, or array of target data. This is the labeled data the model tries to predict; in other words, these are the results of x_train.
  • Epochs: The integer value defines how often the model goes through the training set.
  • Batch size: Integer value that defines the number of samples that will be propagated through the network. After each propagation, the network adjusts the weights of the nodes in each layer.
# Configure the neural network model
model = Sequential()

# Model with n_neurons Neurons
n_neurons = x_train.shape[1] * x_train.shape[2]
print(n_neurons, x_train.shape[1], x_train.shape[2])
model.add(LSTM(n_neurons, return_sequences=True, input_shape=(x_train.shape[1], x_train.shape[2])))
model.add(LSTM(n_neurons, return_sequences=False))
model.add(Dense(25, activation="relu"))
model.add(Dense(1))

# Compile the model
model.compile(optimizer="adam", loss="mean_squared_error")

# Training the model
epochs = 10
early_stop = EarlyStopping(monitor='loss', patience=2, verbose=1)
history = model.fit(x_train, y_train, 
                    batch_size=16, 
                    epochs=epochs, 
                    callbacks=[early_stop])
Epoch 1/10
155/155 [==============================] - 7s 25ms/step - loss: 8.2791e-04
Epoch 2/10
155/155 [==============================] - 4s 24ms/step - loss: 1.3465e-04
Epoch 3/10
155/155 [==============================] - 4s 24ms/step - loss: 1.0998e-04
Epoch 4/10
155/155 [==============================] - 4s 25ms/step - loss: 1.0241e-04
Epoch 5/10
155/155 [==============================] - 4s 24ms/step - loss: 7.4277e-05
Epoch 6/10
155/155 [==============================] - 4s 24ms/step - loss: 6.5786e-05
Epoch 7/10
155/155 [==============================] - 4s 24ms/step - loss: 6.8482e-05
Epoch 8/10
155/155 [==============================] - 4s 24ms/step - loss: 5.0326e-05
Epoch 9/10
155/155 [==============================] - 4s 24ms/step - loss: 4.8574e-05
Epoch 10/10
155/155 [==============================] - 4s 25ms/step - loss: 4.1287e-05

Step #5 Evaluate Model Performance

Next, we validate the model by calculating our predictions’ mean-squared and root-mean-squared errors. However, in time series forecasting metrics can be misleading. It is good to double-check model results using illustrations. Therefore, we plot the input sequences and the forecast to see if our model can continue the time series in a plausible way.

# Get the predicted values
y_pred_scaled = model.predict(x_test)
y_pred = scaler_pred.inverse_transform(y_pred_scaled)
y_test_unscaled = scaler_pred.inverse_transform(y_test.reshape(-1, 1))

# Mean Absolute Error (MAE)
MAE = mean_absolute_error(y_test_unscaled, y_pred)
print(f'Median Absolute Error (MAE): {np.round(MAE, 2)}')

# Mean Absolute Percentage Error (MAPE)
MAPE = np.mean((np.abs(np.subtract(y_test_unscaled, y_pred)/ y_test_unscaled))) * 100
print(f'Mean Absolute Percentage Error (MAPE): {np.round(MAPE, 2)} %')

# Median Absolute Percentage Error (MDAPE)
MDAPE = np.median((np.abs(np.subtract(y_test_unscaled, y_pred)/ y_test_unscaled)) ) * 100
print(f'Median Absolute Percentage Error (MDAPE): {np.round(MDAPE, 2)} %')

# The date from which on the date is displayed
display_start_date = "2018-01-01" 

# Add the difference between the valid and predicted prices
train = pd.DataFrame(train_df[:train_data_length + 1]).rename(columns={'Close': 'x_train'})
valid = pd.DataFrame(train_df[train_data_length:]).rename(columns={'Close': 'y_test'})
valid.insert(1, "y_pred", y_pred, True)
valid.insert(1, "residuals", valid["y_pred"] - valid["y_test"], True)
df_union = pd.concat([train, valid])

# Zoom in to a closer timeframe
df_union_zoom = df_union[df_union.index > display_start_date]

# Create the lineplot
fig, ax1 = plt.subplots(figsize=(16, 8))
plt.title("Predictions vs Ground Truth")
plt.ylabel(stockname, fontsize=18)
sns.despine();
sns.lineplot(data=df_union_zoom, linewidth=1.0, palette='CMRmap', ax=ax1)
plt.show()
Median Absolute Error (MAE): 42.7
Mean Absolute Percentage Error (MAPE): 1.16 %
Median Absolute Percentage Error (MDAPE): 0.92 %
Stock market prediction with neural networks - input data from the S&P 500

At the bottom, we can see the differences between predictions and valid data. Positive values signal that the projections were too optimistic. Negative values mean that the predictions were too pessimistic and that the actual value turned out to be higher than the prediction.

Step #6 Predicting for the Next Week

What can be more satisfying than to see a newly trained model at work? Let’s use our new model to predict next week’s price for the S&P500. We will create a fresh input_sequence with prices from the past N days. Then we scale this input sequence and include it as input in our call to the model.predict() function. 

# Get fresh data until today and create a new dataframe with only the price data

new_df = df

N = sequence_length

# Get the last N steps closing price values and scale the data to be values between 0 and 1
last_N_steps = new_df[-sequence_length:].values
last_N_steps_scaled = mmscaler.transform(last_N_steps)

# Create an empty list and Append past N steps
X_test_new = []
X_test_new.append(last_N_steps_scaled)

# Convert the X_test data set to a numpy array and reshape the data
pred_price_scaled = model.predict(np.array(X_test_new))
pred_price_unscaled = scaler_pred.inverse_transform(pred_price_scaled.reshape(-1, 1))

# Print last price and predicted price for the next week
price_today = np.round(new_df['Close'][-1], 2)
predicted_price = np.round(pred_price_unscaled.ravel()[0], 2)
change_percent = np.round(100 - (price_today * 100)/predicted_price, 2)

plus = '+'; minus = ''
print(f'The close price for {stockname} at {end_date} was {price_today}')
print(f'The predicted close price is {predicted_price} ({plus if change_percent > 0 else minus}{change_percent}%)')
The close price for S&P500 at 2022-05-11 was 4001.05
The predicted close price is 4046.300048828125 (+1.12%)

So for the 9th of April 2020, the model predicts that the S&P500 will close at:

4046.300048828125

Considering today’s (2nd of April 2020) price is 2528 points, our model expects the S&P to gain roughly 124 points in the coming seven days. Of course, this is by no means financial advice. As we have seen before, our model is often wrong.

Summary

This article has shown how to adjust the prediction intervals for a time series forecasting model. We have created a neural network that predicts the price of the S&P500 one week in advance. Finally, we trained and validated the model and made a forecast for the next week.

Varying the input shape is a quick approach to changing the forecasting time steps. However, increasing the length of the time steps also reduces the amount of data we can use for training and testing. In our case, we still have enough data available. But in other cases, where less information is available, this can become a problem. The preferred method is to use a rolling forecast approach or create a multi-output forecast in such a case.

I hope this article was helpful. Should you have questions or remarks, let me know in the comments.

Sources and Further Readings

The links above to Amazon are affiliate links. By buying through these links, you support the Relataly.com blog and help to cover the hosting costs. Using the links does not affect the price.

The post Stock Market Prediction – Adjusting Time Series Prediction Intervals in Python appeared first on relataly.com.

]]> https://www.relataly.com/changing-prediction-intervals-for-time-series-forecasting-models/169/feed/ 2 169