In time series regression, the number of neurons in the final output layer determines how many steps in a time series the model can predict. We can design neural network architectures with multiple outputs in time-series forecasting to forecast more extended periods. Models with one output return single-step forecasts. Models with various outputs can return entire series of time steps and thus deliver a more detailed projection of how a time series could develop in the future. In this tutorial, we create a multi-output multi-step neural network for stock price forecasting using Python and Keras.

This article proceeds as follows: We begin by briefly discussing the architecture of a multi-output neural network. After familiarizing ourselves with the model architecture, we develop a Keras neural network for multi-output regression. For data preparation, we perform various steps, including cleaning, splitting, selecting, and scaling the data. Afterward, we define a model architecture with multiple LSTM layers and ten output neurons in the last layer. This architecture enables the model to generate projections for ten consecutive steps. After configuring the model architecture, we train the model with the historical daily prices of the Apple stock. Finally, we use this model to generate a ten-day forecast.

## Multi-Output Regression vs. Single-Output Regression

In time series regression, we train a statistical model on the past values of a time series to make statements about how the time series develops further. We feed the model with so-called mini-batches and the corresponding target values during model training. The model then creates forecasts for all input batches and compares these predictions to the actual target values to calculate the residuals (prediction errors). In this way, the model can adjust its parameters iteratively and learn to make better predictions.

Next, we will discuss the architecture of a neural network with multiple outputs. The architecture consists of several layers, including an input layer, several hidden layers, and an output layer. The number of neurons in the first layer must match the input data, and the number of neurons in the output layer determines the period length of the predictions. Models with a single neuron in the output layer are used to predict a single time step. It is possible to predict multiple price steps with a single-output model. However, it requires a rolling forecasting approach in which the outputs are iteratively reused to make further-reaching predictions. However, this way is somewhat cumbersome. A more elegant way is to train a multi-output model right away.

A model with multiple neurons in the output layer can predict numerous steps once per batch. Multi-output regression models train on many sequences of subsequent values, followed by the consecutive output sequence. The model architecture thus contains multiple neurons in the initial layer and various neurons in the output layer (as illustrated). In the next part of this tutorial, we will develop a multi-output regression model.

## Implementing a Neural Network Model for Multi-Output Multi-Step Regression in Python

Let’s get started with the hands-on part. In the following, we will develop a Keras neural network that forecasts the Apple stock price. We use historical price data that is available via the yahoo finance API. After obtaining the data via the API, we conduct several steps to prepare and split the data. Finally, we train the neural network.

The code is available on the GitHub repository.

### Prerequisites

Before beginning the coding part, ensure that you have set up your Python 3 environment and required packages. If you don’t have a Python environment, consider Anaconda. To set it up, you can follow the steps in this tutorial.

Also, make sure you install all required packages. In this tutorial, we will be working with the following standard packages:

In addition, we will be using the machine learning libraries Keras, Scikit-learn, and Tensorflow. For visualization, we will be using the Seaborn package.

Please also have either the pandas_datareader or the yfinance package installed. You will use one of these packages to retrieve the historical stock quotes.

You can install these packages using console commands:

*pip install <package name>**conda install <package name>*(if you are using the anaconda packet manager)

### Step #1: Load the Data

The Pandas DataReader library is our first choice for interacting with the yahoo finance API. If the library causes a problem (it sometimes does), you can also use the yfinance package, which should return the same data. We begin by loading historical price quotes of the Apple stock from the public yahoo finance API. Running the code below will load the data into a Pandas DataFrame.

# import pandas_datareader as webreader # Remote data access for pandas import math # Mathematical functions import numpy as np # Fundamental package for scientific computing with Python import pandas as pd # Additional functions for analysing and manipulating data from datetime import date, timedelta, datetime # Date Functions from pandas.plotting import register_matplotlib_converters # This function adds plotting functions for calender dates import matplotlib.pyplot as plt # Important package for visualization - we use this to plot the market data import matplotlib.dates as mdates # Formatting dates from sklearn.metrics import mean_absolute_error, mean_squared_error # Packages for measuring model performance / errors from keras.models import Sequential # Deep learning library, used for neural networks from keras.layers import LSTM, Dense, Dropout # Deep learning classes for recurrent and regular densely-connected layers from keras.callbacks import EarlyStopping # EarlyStopping during model training from sklearn.preprocessing import RobustScaler, MinMaxScaler # This Scaler removes the median and scales the data according to the quantile range to normalize the price data import seaborn as sns # from pandas_datareader.nasdaq_trader import get_nasdaq_symbols # symbols = get_nasdaq_symbols() # Setting the timeframe for the data extraction today = date.today() date_today = today.strftime("%Y-%m-%d") date_start = '2010-01-01' # Getting NASDAQ quotes stockname = 'Apple' symbol = 'AAPL' # df = webreader.DataReader( # symbol, start=date_start, end=date_today, data_source="yahoo" # ) import yfinance as yf #Alternative package if webreader does not work: pip install yfinance df = yf.download(symbol, start=date_start, end=date_today) # # Create a quick overview of the dataset df.head()

The data should comprise the following columns:

- Close
- Open
- High
- Low
- Adj Close
- Volume

The target variable that we are trying to predict is the Closing price (Close).

### Step #2: Explore the Data

Once we have loaded the data, we print a quick overview of the time-series data using different line graphs.

# Plot line charts df_plot = df.copy() ncols = 2 nrows = int(round(df_plot.shape[1] / ncols, 0)) fig, ax = plt.subplots(nrows=nrows, ncols=ncols, sharex=True, figsize=(14, 7)) for i, ax in enumerate(fig.axes): sns.lineplot(data = df_plot.iloc[:, i], ax=ax) ax.tick_params(axis="x", rotation=30, labelsize=10, length=0) ax.xaxis.set_major_locator(mdates.AutoDateLocator()) fig.tight_layout() plt.show()

The line plots look as expected and reflect the Apple stock price history.

### Step #3: Preprocess the Data

Next, we prepare the data for the training process. Preparing the data for a multivariate regression model involves several steps:

- Selecting features for model training
- Scaling and splitting the data into training and testing
- Slicing the time series into several shifted training batches

We begin by creating a copy of the initial data and resetting the index.

# Indexing Batches df_train = df.sort_values(by=['Date']).copy() # We safe a copy of the dates index, before we need to reset it to numbers date_index = df_train.index # We reset the index, so we can convert the date-index to a number-index df_train = df_train.reset_index(drop=True).copy() df_train.head(5)

We proceed with feature selection. To keep things simple, we will use the features from the input data without any modifications. After selecting the features, we scale them to a range between 0 and 1. To ease unscaling the predictions after training, we create two different scalers: One for the training data, which takes five columns, and one for the output data that scales a single column (the Close Price). I have covered feature engineering in a separate article if you want to learn more about this topic.

def prepare_data(df): # List of considered Features FEATURES = ['Open', 'High', 'Low', 'Close', 'Volume'] print('FEATURE LIST') print([f for f in FEATURES]) # Create the dataset with features and filter the data to the list of FEATURES df_filter = df[FEATURES] # Convert the data to numpy values np_filter_unscaled = np.array(df_filter) #np_filter_unscaled = np.reshape(np_unscaled, (df_filter.shape[0], -1)) print(np_filter_unscaled.shape) np_c_unscaled = np.array(df['Close']).reshape(-1, 1) return np_filter_unscaled, np_c_unscaled np_filter_unscaled, np_c_unscaled = prepare_data(df_train) # Creating a separate scaler that works on a single column for scaling predictions # Scale each feature to a range between 0 and 1 scaler_train = MinMaxScaler() np_scaled = scaler_train.fit_transform(np_filter_unscaled) # Create a separate scaler for a single column scaler_pred = MinMaxScaler() np_scaled_c = scaler_pred.fit_transform(np_c_unscaled)

The final step of the data preparation is to create the structure for the input data. This structure needs to match the input layer of the model architecture.

Running the code below starts a sliding window script that cuts the initial time series data into multiple slices, i.e., mini-batches. Each batch is a smaller fraction of the initial time series shifted by a single step. Because we will feed our model with multivariate input data, the time series consists of five input columns/features. Each batch comprises a period of 50 steps from the time series and an output sequence of ten consecutive values. To validate that the batches have the right shape, we visualize mini-batches in a line graph with their consecutive target values.

# Set the input_sequence_length length - this is the timeframe used to make a single prediction input_sequence_length = 50 # The output sequence length is the number of steps that the neural network predicts output_sequence_length = 10 # # Prediction Index index_Close = df_train.columns.get_loc("Close") # Split the training data into train and train data sets # As a first step, we get the number of rows to train the model on 80% of the data train_data_length = math.ceil(np_scaled.shape[0] * 0.8) # Create the training and test data train_data = np_scaled[:train_data_length, :] test_data = np_scaled[train_data_length - input_sequence_length:, :] # The RNN needs data with the format of [samples, time steps, features] # Here, we create N samples, input_sequence_length time steps per sample, and f features def partition_dataset(input_sequence_length, output_sequence_length, data): x, y = [], [] data_len = data.shape[0] for i in range(input_sequence_length, data_len - output_sequence_length): x.append(data[i-input_sequence_length:i,:]) #contains input_sequence_length values 0-input_sequence_length * columns y.append(data[i:i + output_sequence_length, index_Close]) #contains the prediction values for validation (3rd column = Close), for single-step prediction # Convert the x and y to numpy arrays x = np.array(x) y = np.array(y) return x, y # Generate training data and test data x_train, y_train = partition_dataset(input_sequence_length, output_sequence_length, train_data) x_test, y_test = partition_dataset(input_sequence_length, output_sequence_length, test_data) # Print the shapes: the result is: (rows, training_sequence, features) (prediction value, ) print(x_train.shape, y_train.shape) print(x_test.shape, y_test.shape) # Validate that the prediction value and the input match up # The last close price of the second input sample should equal the first prediction value nrows = 3 # number of shifted plots fig, ax = plt.subplots(nrows=nrows, ncols=1, figsize=(16, 8)) for i, ax in enumerate(fig.axes): xtrain = pd.DataFrame(x_train[i][:,index_Close], columns={f'x_train_{i}'}) ytrain = pd.DataFrame(y_train[i][:output_sequence_length-1], columns={f'y_train_{i}'}) ytrain.index = np.arange(input_sequence_length, input_sequence_length + output_sequence_length-1) xtrain_ = pd.concat([xtrain, ytrain[:1].rename(columns={ytrain.columns[0]:xtrain.columns[0]})]) df_merge = pd.concat([xtrain_, ytrain]) sns.lineplot(data = df_merge, ax=ax) plt.show

### Step #3: Prepare the Neural Network Architecture and Train the Multi-Output Regression Model

Now that we have the training data prepared and ready, the next step is to configure the architecture of the multi-out neural network. Because we will be using multiple input series, our model is, in fact, a multivariate architecture so that it corresponds to the input training batches.

We choose a comparably simple architecture with only two LSTM layers and two additional dense layers. The first dense layer has 20 neurons, and the second layer is the output layer, which has ten output neurons. If you wonder how I got to the number of neurons in the third layer, I conducted several experiments and found that this number leads to solid results.

To ensure that the architecture matches our input data’s structure, we reuse the variables for the previous code section (n_input_neurons, n_output_neurons. The input sequence length is 50, and the output sequence (the steps for the period we want to predict) is ten.

# Configure the neural network model model = Sequential() n_output_neurons = output_sequence_length # Model with n_neurons = inputshape Timestamps, each with x_train.shape[2] variables n_input_neurons = x_train.shape[1] * x_train.shape[2] print(n_input_neurons, x_train.shape[1], x_train.shape[2]) model.add(LSTM(n_input_neurons, return_sequences=True, input_shape=(x_train.shape[1], x_train.shape[2]))) model.add(LSTM(n_input_neurons, return_sequences=False)) model.add(Dense(20)) model.add(Dense(n_output_neurons)) # Compile the model model.compile(optimizer='adam', loss='mse')

After configuring the model architecture, we can initiate the training process and illustrate how the loss develops over the training epochs.

# Training the model epochs = 10 batch_size = 16 early_stop = EarlyStopping(monitor='loss', patience=5, verbose=1) history = model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_data=(x_test, y_test) ) #callbacks=[early_stop])

# Plot training & validation loss values fig, ax = plt.subplots(figsize=(10, 5), sharex=True) plt.plot(history.history["loss"]) plt.title("Model loss") plt.ylabel("Loss") plt.xlabel("Epoch") ax.xaxis.set_major_locator(plt.MaxNLocator(epochs)) plt.legend(["Train", "Test"], loc="upper left") plt.grid() plt.show()

### Step #5 Evaluate Model Performance

Now that we have trained the model, we can make forecasts on the test data and use traditional regression metrics such as the MAE, MAPE, or MDAPE to measure the performance of our model.

# Get the predicted values y_pred_scaled = model.predict(x_test) # Unscale the predicted values y_pred = scaler_pred.inverse_transform(y_pred_scaled) y_test_unscaled = scaler_pred.inverse_transform(y_test).reshape(-1, output_sequence_length) # Mean Absolute Error (MAE) MAE = mean_absolute_error(y_test_unscaled, y_pred) print(f'Median Absolute Error (MAE): {np.round(MAE, 2)}') # Mean Absolute Percentage Error (MAPE) MAPE = np.mean((np.abs(np.subtract(y_test_unscaled, y_pred)/ y_test_unscaled))) * 100 print(f'Mean Absolute Percentage Error (MAPE): {np.round(MAPE, 2)} %') # Median Absolute Percentage Error (MDAPE) MDAPE = np.median((np.abs(np.subtract(y_test_unscaled, y_pred)/ y_test_unscaled)) ) * 100 print(f'Median Absolute Percentage Error (MDAPE): {np.round(MDAPE, 2)} %') def prepare_df(i, x, y, y_pred_unscaled): # Undo the scaling on x, reshape the testset into a one-dimensional array, so that it fits to the pred scaler x_test_unscaled_df = pd.DataFrame(scaler_pred.inverse_transform((x[i]))[:,index_Close]).rename(columns={0:'x_test'}) y_test_unscaled_df = [] # Undo the scaling on y if type(y) == np.ndarray: y_test_unscaled_df = pd.DataFrame(scaler_pred.inverse_transform(y)[i]).rename(columns={0:'y_test'}) # Create a dataframe for the y_pred at position i, y_pred is already unscaled y_pred_df = pd.DataFrame(y_pred_unscaled[i]).rename(columns={0:'y_pred'}) return x_test_unscaled_df, y_pred_df, y_test_unscaled_df def plot_multi_test_forecast(x_test_unscaled_df, y_test_unscaled_df, y_pred_df, title): # Package y_pred_unscaled and y_test_unscaled into a dataframe with columns pred and true if type(y_test_unscaled_df) == pd.core.frame.DataFrame: df_merge = y_pred_df.join(y_test_unscaled_df, how='left') else: df_merge = y_pred_df.copy() # Merge the dataframes df_merge_ = pd.concat([x_test_unscaled_df, df_merge]).reset_index(drop=True) # Plot the linecharts fig, ax = plt.subplots(figsize=(20, 8)) plt.title(title, fontsize=12) ax.set(ylabel = stockname + "_stock_price_quotes") sns.lineplot(data = df_merge_, linewidth=2.0, ax=ax) # Creates a linechart for a specific test batch_number and corresponding test predictions batch_number = 50 x_test_unscaled_df, y_pred_df, y_test_unscaled_df = prepare_df(i, x_test, y_test, y_pred) title = f"Predictions vs y_test - test batch number {batch_number}" plot_multi_test_forecast(x_test_unscaled_df, y_test_unscaled_df, y_pred_df, title)

The quality of the predictions is acceptable, considering the goal of this architecture was not to create an excellent model. However, there is certainly room for improvement.

### Step #6 Create a New Forecast

Finally, let’s create a forecast on a new dataset. We take the scaled dataset from section 2 (np_scaled) and extract a series with the latest 50 values. Then we use these values to generate a new prediction for the next ten days. We visualize the multi-step forecast in another line chart.

# Get the latest input batch from the test dataset, which is contains the price values for the last ten trading days x_test_latest_batch = np_scaled[-51:-1,:].reshape(1,50,5) # Predict on the batch y_pred_scaled = model.predict(x_test_latest_batch) y_pred_unscaled = scaler_pred.inverse_transform(y_pred_scaled) # Prepare the data and plot the input data and the predictions x_test_unscaled_df, y_test_unscaled_df, _ = prepare_df(0, x_test_latest_batch, '', y_pred_unscaled) plot_multi_test_forecast(x_test_unscaled_df, '', y_pred_df, "x_new Vs. y_new_pred")

## Summary

This tutorial has shown how we can use multiple output neural networks to make predictions over different time steps. We discussed the necessary architecture of a recurrent neural network and learned how to process the data accordingly. In addition, we trained a multi-output regression model that predicts ten price steps for Apple stock. Finally, we visualized the multi-step predictions of this model in a line plot.

Feel free to test other hyperparameters and adjust the model architecture. Adding more neurons to the output layers will increase the prediction horizon. But keep in mind that prediction error will increase with the prediction horizon.

I hope this article was helpful in understanding multi-output neural networks better. If you have any questions or comments, please let me know.

## Sources and Further Reading

- Charu C. Aggarwal (2018) Neural Networks and Deep Learning
- Jansen (2020) Machine Learning for Algorithmic Trading: Predictive models to extract signals from market and alternative data for systematic trading strategies with Python
- Aurélien Géron (2019) Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems
- David Forsyth (2019) Applied Machine Learning Springer
- Andriy Burkov (2020) Machine Learning Engineering

*The links above to Amazon are affiliate links. By buying through these links, you support the Relataly.com blog and help to cover the hosting costs. Using the links does not affect the price.*

6) Once I train a model with for example EURUSD data and saved the model, can I use it to predict GOLD or I have to create and train another model with GOLD data? If I would have to create a separate model to train, how then can a single model be used on multiple assets for real time (real life predictions)?

I have come to say thank you for taking out time to look into my notebook and for those suggestions, the prediction shape is now is order (2 dimensional). Apparently my last LSTM layer had return_sequences set to true which was supplying a 3D to the output layer. I however have some questions:

1). You mentioned in your blog that the number of units in the first LSTM layer should be the input size. Can you please explain why?

2) Having trained my model with 23 variables, do i have to predict also real time data with 23 variables or this can be changed when re-shaping the new data before prediction?

3). Do you have a code snippet for streaming real-time price data from Oanda? yfinance does not give real-time data for currency pairs.

4) What do you think about including the prices data as input to the model? Is prices with indicator data better than with indicators alone as input variables?

5) Also, how do you advise to get the best number of hidden layers for a model. I have tried using the for-loop way but it was giving me issues with the output dimension.

I took another look at your notebook. Although I didn’t find the error, I denoted a couple of things that might help you find it.

I had to switch from Keras to Tensorflow. Keras – otherwise, my GPU won’t work. You can do the same by changing the imports from Keras to Tensorflow.Keras. Then imports look like this:from tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Dense, LSTM, Dropout, GRU, Bidirectional, Activation

I was surprised by the output format for the predictions after hyperparameter tuning: (630, 50, 1). Based on how you scaled the data, I would expect the predictions to be in the format (630, 1).

It is crucial that the predictions are in the same shape as the data that you used to fit the scaler. So if you create a scaler for a two-dimensional array, ensure that your later predictions are also 2-dimensional.The cause for the strange prediction shape might be a problem with the hyperparameter tuning where you automatically create architectures with different layers. Maybe the algorithm messes something up with the final layer, which should be dense(1) for a single-step prediction.

In general, don’t waste too much time on hyperparameter, before your model delivers realistic prediction values. Reduce complexity to speed up training times and test more things in shorter time.

I hope this helps you find the error!

Florian

The prediction shape has been an headache really. I will use tensorflow.keras as advised and also manually create architectures with different layers and update you soon.

Thank you so much for this sir, pls how can i reach you privately? Its very critical to my success and eventual graduating from school. My email is folafoyeg@gmail.com

Hi Olaitan

Thank you for your kind feedback.

You can send me a message to flomue@relataly.com or contact me on LinkedIn.

Cheers

Florian

Thank you for your response. I sent you a mail since that day but I am yet to get a response. Please help, time is against me.

I took a quick look at your code. You are currently using 28 different features at the same time.

Have you tried to lower the model complexity and train it with only one feature?

How do the results change?

Also, you have some hyperparameter tuning in your notebook. My tip is to remove all optimizations until you have a baseline model that gets some okisch results.

I try to run the notebook later today, but I am currently a bit short on time.

Thank you for your response and tips sir. I have been able to get the model to predict (with the 28 variables and optimizations in place). I will send you the updated code file now. Its the result interpretation and graph plotting that i can’t get right because the prediction result is in 3D.

This is a very useful content, it has help me in understanding multistep forcasting. How can I privately reachout to you please?

hey there ,

thank you for your work it has been so helpful !

my quesetion is how to change the forecasting horizons ?

Hey Shim, you can change the forecasting horizon by either increasing the number of ouputs (variable: output_sequence_length). Hope this helps

Hi

Great post!

A quick question, what is the best idea in multi-output regression when samples have different output lengths.

Thanks

So, first, this IS a useful article how to work with data. However, it seems to be a terrible architecture design for the task at hand. There needs to be a lot more custom functions in the neural design not just “oh use LSTM”

Thanks for sharing your experience with the synthetic data! Just as with single outputs it’s difficult to say in advance which parameters will lead to good results. There is often no way around conducting experiments. Two things you could try are modifying the hidden layers and the length of the input periods. I’ll also experiment with the data as soon as I get back from vacation. 🙂

hi, I ran this code using your Sine function from an earlier example. So I imported the date as per your code (from Yahoo) and then below that I added:

steps = df.shape[0]

gradient = 0.02

list_a = []

for i in range(0, steps, 1):

y = round(gradient * i + math.sin(math.pi * 0.125 * i), 5)

list_a.append(y)

df2 = pd.DataFrame({“Close”: list_a,

“Open”: list_a,

“High”: list_a,

“Low”: list_a,

“Volume”: list_a,

“Adj Close”: list_a},

columns=[“Close”,”Open”,”High”,”Low”,”Volume”,”Adj Close”])

df2.index = df.index

df = df2

so the original dataframe df from Yahoo is now filled with artificial data keeping the same dates. The results are not very good if you compare it to your example given here: https://www.relataly.com/stock-market-prediction-using-multivariate-time-series-in-python/1815/

Even if I increase the number of epochs. Is this to be expected when using multiple nodes in the output layer?

thanks, I just found this 1 just now. I already implemented 2 of your examples. Will implement this one as well!

Hello Ed, were you able to successfully implemet this?

hi, I implemented it inside Amibroker. But I found the results poor even with artificial data. So my plan is to work on “reinforcement learning” There are examples on the net. The difference is that you do not intend to predict the price ito the future but you learn the neural network to trade. I hope Florian will look into this as well and make an example So here is 1 of Florian’s codes implemented inside Amibroker:

https://forum.amibroker.com/t/supervised-learning-example/27990

But on reinforcement learning there are also many examples on the net. I still have been too lazy to implement 1.

Thank you for your response, I would check out Amibroker.

Hello Ed, i trust you are doing great? Please how can i contact you privately, I need some help. My email is folafoyeg@gmail.com. I look forward to hearing from you soon.

no, i think that would be the blind trying to help the blind. I am able to use existing Python code examples and build it inside Amibroker. But the examples given on relataly.com are based on “supervised learning”. If you have a periodic time series this works great but with futures data, as far as I have been able to tell, it can not reliable say if the next price bar will be up or down. But I did not do the hard work yet really digging into the nuts and bolts. So I am not able yet to play around with it. But my first impression is this will not work.

So therefor I have now implemented a “reinforcement learning” example I found on the net. I just was able to implement it in Amibroker but I am not yet at a stage to say if I can do anything useful with it. I need to put in the hard work. Same for you I think. You have to do the work yourself. The reinforcement learning example I implemented you can find here: https://github.com/WANGXinyiLinda/Policy-Gradient-Trading-Algorithm-by-Maximizing-Sharpe-Ratio/tree/master/PG

But it is quite complicated. I have it working but now I have to go through the code to understand what each parameter does in order to add additional input parameters and play around with the variables. So far it seems that again a periodic time series gives great results but real data not so much (yet). But I just used the price. I have some additional data that probably could improve the results. If there are patterns in the data the NN should be able to find them.