Affinity propagation is an unsupervised clustering technique that stands out from other clustering approaches by its capacity to determine the number of clusters in a dataset. This tutorial demonstrates this capacity by applying the technique to analyze the crypto market structure. We perform a cluster analysis of historical prices of cryptocurrencies (such as Bitcoin and Ethereum) and group those cryptos into clusters that have shown similar price fluctuations during the past 50 days. The result is a crypto market map that can help to assess portfolio risks.
The rest of this article is organized as follows: First, we look at some concepts behind market structuring, such as covariance, lasso regression, and clustering. The second part is a Python hands-con tutorial, in which we combine affinity propagation with partial covariance to visualize the crypto market on a 2D plane and in a 3D space.
The Problem with Prototype-based Clustering
Clustering is an unsupervised learning technique that groups similar objects into clusters and separates them from different ones. One of the most popular clustering techniques is k-means. K-means belongs to the so-called prototype-based clustering techniques, which separate data points into a predefined number of groups (in the case of k-means, the groups are of equal variance).
The prototype-based clustering approach works great if the number of clusters in a dataset is known and the clusters have similar despair. However, when we deal with real-world problems, we often encounter more complex data for which the optimal number of clusters is unknown and difficult or even impossible to guess. In such a case, affinity propagation has a significant advantage because it can automatically estimate the number of clusters.
Brief Overview of How Affinity Propagation Works
The idea of affinity propagation is to identify clusters by measuring the similarity of data points relative to one another. The algorithm chooses data points as cluster centers that best represent other data points near them.
We can imagine the process of identifying these representative data points as an election. Each data point (i) is a voter who casts votes and a candidate (k) who can receive votes from other voters. Votes are a measure of the similarity of data points. A voter who gives many votes to a candidate expresses that this data point is similar to him and therefore is suitable for representing him as a cluster center. The voting process continues until the algorithm reaches a consensus and selects a set number of cluster candidates.
The clustering process involves many separate steps (This article provides a detailed description of the steps involved) and works with several matrices:
- The similarity matrix assesses the suitability of data points (candidates) to act as the cluster centers.
- The availability matrix (or responsibility matrix) collects the support of the data points for the candidates (potential cluster centers) and their suitability to represent them.
- The criterion matrix sums up the results and defines the clusters. Data points with equal scores in the criterion matrix are considered part of the same cluster.
Visualizing Cryptocurrency Market Structures in Python
Now that we have an overview of affinity propagation, we can implement it in Python. Our goal is to analyze the crypto market structure and create a visual representation of price similarity. We do not specify the number of clusters and let the algorithm detect them.
In the following, we carry out the following steps:
- Define a list of cryptocurrencies and download their price data via cmcscraper from coinmarketcap.
- Visualize the historical price data to understand the data better.
- Prepare and clean the data before clustering them.
- Use Affinity Propagation to cluster the cryptocurrencies.
- Calculate the covariance matrix between clusters.
- Arrange the cryptocurrencies on a 2D map into clusters and create a network overlay based on covariance.
- Briefly interpret the crypto market map.
The Python code for this tutorial is available in the relataly repository on GitHub.
Prerequisites
Before beginning with the coding part, ensure that you have set up your Python 3 environment and required packages. Consider Anaconda if you don’t have a Python environment set up yet. To set it up, you can follow the steps in this tutorial.
Also, make sure you install all required packages. In this tutorial, we will be working with the following standard packages:
Please also make sure you have the Cmcscaper package installed. We will be using it to download past crypto prices from coinmarketcap.
You can install these packages using console commands:
- pip install <package name>
- conda install <package name> (if you are using the anaconda packet manager)
Step #1: Load the Stock Market Data
We start by loading historical crypto price data from Coinmarketcap. To download the data, we use Cmcscraper, a Python library that allows us to collect Coinmarketcap data without signing up for the official API.
The download returns a dataframe with daily price quotes (Close, Open, Avg) for cryptocurrencies between 2016 and today. You can use the dictionary (“symbol_dict”) to control which cryptos you want to include in the data. We limit the data we use in our cluster analysis to the last 50 days. In this way, we let the correlation take earlier price developments into account. But, it’s up to you to specify a different period. In addition, instead of using absolute price values, we will use the daily percentage fluctuations.
Loading the data can take several minutes, depending on how many cryptocurrencies we include in the request. So it makes sense not to load the data every time you run the code. Therefore, the code below stores the historical prices in a CSV file. If you run the code below, the script will check if the data already exists. If it does, it will use the data from the CSV file. Otherwise, it will load a fresh copy of the data from coinmarketcap.
# A tutorial for this file is available at www.relataly.com
from cryptocmd import CmcScraper
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from sklearn import cluster, covariance, manifold
# This list defines the coins that will be considered
symbol_dict = {
'BTC': 'Bitcoin',
'ETH': 'Ethereum',
'XRP': 'Ripple',
'ADA': 'Cardano',
'KMD': 'Komodo',
'BNB': 'Binance Coin',
'DOGE': 'Doge Coin',
'LTC': 'Litecoin',
'USDT': 'Tether',
'ZRX': 'Zer0',
'BAT': 'Battoken',
'UNI': 'DOT',
'PSG': 'PSG Token',
'ACM': 'ACM Token',
'RSR': 'AS Rom Token',
'JUV': 'Juventus Turin Token',
'ATM': 'Atletico Madrid Token',
'SOL': 'Solana',
'MATIC': 'Polygon',
'LINK': 'Link',
'ETC': 'Ethereum Classic',
'AVAX': 'Avalance',
'DCR': 'Decred',
'WAVES': 'WAVES',
'VET': 'Vechain',
'ARK': 'ARK',
'BCH': 'Bitcoin Cash',
'ICP': 'Internet Computer',
'DGB': 'Digibyte',
'BTT': 'BitTorrent',
'CEL': 'Celsius',
'SNX': 'Synthetix',
'ENJ': 'Enjin',
'ZIL': 'Zilliqa',
'CHZ': 'Chilliz',
'THETA': 'Theta',
'XLM': 'Stellar Lumen',
'SYS': 'Sys Coin',
'LRC': 'Loopring',
'RLC': 'LRC',
'EOS': 'EOS',
'NEO': 'NEO',
'MIOTA': 'IOTA',
'CAKE': 'Cake Defi',
'BLZ': 'BLZ',
'XMR': 'Monero',
'FORTH': 'Ampleforth'
}
# Download historic crypto prices via CmcScraper
def load_fresh_data_and_save_to_disc(symbol_dict, save_path):
# Initialize Coin Symbols List
symbols, names = np.array(sorted(symbol_dict.items())).T
for symbol in symbols:
# Initialise scraper without time interval
scraper = CmcScraper(symbol)
# Pandas dataFrame for the same data
df_coin_prices = scraper.get_dataframe()
df = pd.DataFrame()
print(f'fetching prices for {symbol}')
df[symbol + '_Open'] = df_coin_prices['Open']
df[symbol + '_Close'] = df_coin_prices['Close']
df[symbol + '_Avg'] = (df_coin_prices['Close'] + df_coin_prices['Open']) / 2
# Daily price fluctuations in percent
df[symbol + '_p'] = (df_coin_prices['Open'] - df_coin_prices['Close']) / df_coin_prices['Open']
if symbol == symbols[0]:
# Create a new DataFrame for the first cryptocurrency in the list
df_crypto = df.copy()
else:
# Merge the new price data with the existing DataFrame
df_crypto = pd.merge(
left=df_crypto, right=df, how="outer",
left_index=True, right_index=True)
# Remove all cryptocurrencies that
filter_columns = [s for s in df_crypto.columns if '_p' in s]
X_df_filtered = df_crypto[filter_columns].copy()
X_df_filtered.to_csv(save_path + 'historical_crypto_prices.csv')
return names, symbols, X_df_filtered
save_path = '' # save_path
# If set to False the data will only be downloaded when you execute the code
# Set to True, if you want a fresh copy of the data.
new_data = True
if new_data == False:
try:
print('loading from disk')
X_df_filtered = pd.read_csv(save_path + 'historical_crypto_prices.csv')
if 'Unnamed: 0' in X_df_filtered.columns:
X_df_filtered = X_df_filtered.drop(['Unnamed: 0'], axis=1)
symbols, names = np.array(sorted(symbol_dict.items())).T
print(list(X_df_filtered.columns))
except:
print('no existing price data found - loading fresh data from coinmarketcap and saving them to disk')
names, symbols, X_df_filtered = load_fresh_data_and_save_to_disc(symbol_dict, save_path)
print(list(symbols))
else:
print('loading fresh data from coinmarketcap and saving them to disk')
names, symbols, X_df_filtered = load_fresh_data_and_save_to_disc(symbol_dict, save_path)
print(list(symbols))
# Limit the price data to the last t days
t= 50 # in days
X_df_filtered = X_df_filtered[:t]
X_df_filtered.head()
The data looks good, so let’s continue.
Step #2 Plotting Crypto Price Charts
Now that the data is available, we can visualize it in various line graphs. The visualization helps us better understand what kind of data we are dealing with and check if the download was successful.
# Create Prices Charts for all Cryptocurrencies
list_length = X_df_filtered.shape[1]
ncols = 10
nrows = int(round(list_length / ncols, 0))
height = list_length/3 if list_length > 30 else 4
fig, axs = plt.subplots(nrows=nrows, ncols=ncols, sharex=True, sharey=True, figsize=(20, height))
for i, ax in enumerate(fig.axes):
if i < list_length:
sns.lineplot(data=X_df_filtered, x=X_df_filtered.index, y=X_df_filtered.iloc[:, i], ax=ax)
ax.set_title(X_df_filtered.columns[i])
plt.show()
Everything looks as expected.
Step #3 Clustering Cryptocurrencies using Affinity Propagation
Next, we must prepare the data and run the affinity propagation algorithm. For some cryptocurrencies, we may encounter data that contains NaN values. Because clustering is sensitive to missing values, we must ensure good data quality. In addition, the Python code below will convert the DataFrame into a NumPy array and transpose it into a form where we have crypto assets as records and the days as columns.
Running the code below returns a dictionary of clusters and the cryptocurrencies that the affinity propagation algorithm has assigned to them.
# Drop NaN values
X_df = pd.DataFrame(np.array(X_df_filtered)).dropna()
# Transpose the data to structure prices along columns
X = X_df.copy()
X /= X.std(axis=0)
X = np.array(X)
# Define an edge model based on covariance
edge_model = covariance.GraphicalLassoCV()
# Standardize the time series
edge_model.fit(X)
# Group cryptos to clusters using affinity propagation
# The number of clusters will be determined by the algorithm
cluster_centers_indices , labels = cluster.affinity_propagation(edge_model.covariance_, random_state=1)
cluster_dict = {}
n_labels = labels.max()
print(f"{n_labels} Clusters")
for i in range(n_labels + 1):
clusters = ', '.join(names[labels == i])
print('Cluster %i: %s' % ((i + 1), clusters))
cluster_dict[i] = (clusters)
We can see that the algorithm has identified 13 different clusters in the data and that there are a couple of clusters with only a single member. You will most likely encounter different results depending on when you run it.
Step #4 Create a 2D Positioning Model based on the Graph Structure
In addition to clusters, we also want to show the covariance between cryptocurrencies in our Crypto Market map. We need a graph-like structure that contains the covariance and position data of the cryptocurrencies for each crypto pair.
In addition, we use a node position model that calculates their relative position on a 2D plane from the covariance of the cryptocurrencies. However, the position is only relative to each other, so the absolute axes have no meaning.
# Create a node_position_model that find the best position of the cryptos on a 2D plane
# The number of components defines the dimensions in which the nodes will be positioned
node_position_model = manifold.LocallyLinearEmbedding(n_components=2, eigen_solver='dense', n_neighbors=20)
embedding = node_position_model.fit_transform(X.T).T
# The result are x and y coordindates for all cryptocurrencies
pd.DataFrame(embedding)
# Create an edge_model that represents the partial correlations between the nodes
partial_correlations = edge_model.precision_.copy()
d = 1 / np.sqrt(np.diag(partial_correlations))
partial_correlations *= d
partial_correlations *= d[:, np.newaxis]
# Only consider partial correlations above a specific threshold (0.02)
non_zero = (np.abs(np.triu(partial_correlations, k=1)) > 0.02)
# Convert the Positioning Model into a DataFrame
data = pd.DataFrame.from_dict({"embedding_x":embedding[0],"embedding_y":embedding[1]})
# Add the labels to the 2D positioning model
data["labels"] = labels
print(data.shape)
data.head()
The next step is to create a graph of the partial correlations.
Step #5 Visualize the Crypto Market Structure
Our goal is to visualize differences in the covariance between crypto pairs by varying the connection strengths. We calculate the line strength by normalizing the covariance of the crypto pairs. In addition, we visualize the distribution of the covariance.
# Create an array with the segments for connecting the data points start_idx, end_idx = np.where(non_zero) segments = [[np.array([embedding[:, start], embedding[:, stop]]).T, start, stop] for start, stop in zip(start_idx, end_idx)] # Create a normalized representation of partial correlation between crypto currencies # We can later use covariance to vizualize the strength of the connections pc = np.abs(partial_correlations[non_zero]) normalized = (pc-min(pc))/(max(pc)-min(pc)) # plot the distribution of covariance between the cryptocurrencies sns.histplot(pc)
The hist plot shows that the covariance between the crypto pairs is mostly below 0.005.
Finally, it is time to map cryptocurrencies on a 2D plane. To do this, we first define the cryptocurrencies using their relative position data with a scatterplot. We set the color of the points based on their clusters so that points in the same cluster are colored the same. Subsequently, we connect the points to the data from the edge model. The covariance between the crypto pairs determines the strength of their connections.
We also define the color of the connections as follows.
- The map only shows connections with a covariance greater than 0.002.
- Connections with a covariance greater than 0.05 are colored red.
- Otherwise, connections between points within a cluster are shown in the cluster’s color.
- We color connections in grey that are between points of different clusters.
Last but not least, we add the labels of the cryptocurrencies.
# Visualization
plt.figure(1, facecolor='w', figsize=(20, 8))
plt.clf()
ax = plt.axes([0., 0., 1., 1.])
# Plot the nodes using the coordinates of our embedding
sc = sns.scatterplot(data=data, x="embedding_x", y="embedding_y", zorder=1, s = 350 * d ** 2, c=labels, cmap=plt.cm.nipy_spectral, alpha=.9, palette="muted")
# Plot the covariance edges between the nodes (scatter points)
line_strength = 3.2
for index, ((x, y), start, stop) in enumerate(segments):
norm_partial_correlation = normalized[index]
if list(data.iloc[[start]]['labels'])[0] == list(data.iloc[[stop]]['labels'])[0]:
if norm_partial_correlation > 0.5:
color = 'red'; linestyle='solid'
else:
color = plt.cm.nipy_spectral(list(data.iloc[[start]]['labels'])[0] / float(n_labels)); linestyle='solid'
else:
if norm_partial_correlation > 0.5:
color = 'red'; linestyle='solid'
else:
color = 'grey'; linestyle='dashed'
plt.plot(x, y, alpha=.4, zorder=0, linewidth=normalized[index]*line_strength, color=color, linestyle=linestyle)
# Labels the nodes and position the labels to avoid overlap with other labels
for index, (name, label, (x, y)) in enumerate(zip(names, labels, embedding.T)):
dx = x - embedding[0]
dy = y - embedding[1]
dy[index], dx[index] = 1, 1
this_dx = dx[np.argmin(np.abs(dy))]
this_dy = dy[np.argmin(np.abs(dx))]
if this_dx > 0:
horizontalalignment = 'left'
x = x + .005
else:
horizontalalignment = 'right'
x = x - .004
if this_dy > 0:
verticalalignment = 'bottom'
y = y + .01
else:
verticalalignment = 'top'
y = y + .01
plt.text(x, y, name, size=10,
horizontalalignment=horizontalalignment,
verticalalignment=verticalalignment)
# Label customization options
# color = plt.cm.nipy_spectral(label / float(n_labels))
# bbox=dict(facecolor=plt.cm.nipy_spectral(label / float(n_labels)), edgecolor="w", alpha=.2)
plt.axis('off')
plt.show()
Note that you will likely see a different map when you run the code on your machine. Differences result from changes in market prices and covariance that lead to different graph structures.
Let’s see what the crypto market map tells us.
Interpretation
The 2D crypto market map tells us several things:
- Most cryptos fall into the light green and the dark green clusters that correspond to different types of crypto (Decentralized Finance Coins, NFT/Metaverse Coins).
- There is a significant covariance between large-cap players in the crypto space, such as Cardano and Loopring and Ethereum and Bitcoin, which is plausible considering recent price movements. Some results are surprising, for example, the partial correlation between NEO and Ethereum Classic.
- Some clusters are isolated and contain only a single member, for example, Tether, Komodo, AC Milan token, Wave token, and Dogecoin). The reason is that the prices of these coins/tokens have developed independently of the market.
- Tether is a stable coin that does not change in price. It, therefore, strongly differs from the other cryptocurrencies on our map.
- Komodo has been trading sideways without following the general market trend.
- And the MCM token is a soccer token that has recently outperformed the market.
- Soccer tokens are shown in dark blue. The prices of these tokens correlate with the performance of soccer clubs and the development of the soccer season. It, therefore, makes perfect sense that these tokens are grouped into a cluster. An exception is the AC Milan token, which recently performed better than the other soccer tokens.
Step #6 Creating a 3D Representation
Instead of a 2D representation of the data points, we can also use a 3D node positioning model. For this purpose, the node positioning model distributes the affinity values over three dimensions.
# Find the best position of the cryptos on a 3D plane
node_position_model = manifold.LocallyLinearEmbedding(n_components=3, eigen_solver='dense', n_neighbors=20)
embedding = node_position_model.fit_transform(X.T).T
# The result are x and y coordindates for all cryptocurrencies
pd.DataFrame(embedding)
# Display a graph of the partial correlations
partial_correlations = edge_model.precision_.copy()
d = 1 / np.sqrt(np.diag(partial_correlations))
partial_correlations *= d
partial_correlations *= d[:, np.newaxis]
non_zero = (np.abs(np.triu(partial_correlations, k=1)) > 0.02)
data = pd.DataFrame.from_dict({"embedding_x":embedding[0],"embedding_y":embedding[1],"embedding_z":embedding[1]})
data["labels"] = labels
data["names"] = names
import matplotlib.pyplot as plt
import numpy as np
fig = plt.figure(figsize=(20,20))
ax = fig.add_subplot(projection='3d')
xs = data["embedding_x"]
ys = data["embedding_y"]
zs = data["embedding_z"]
sc = ax.scatter(xs, ys, zs, c=labels, s=100)
for i in range(len(data)):
x = xs[i]
y = ys[i]
z = zs[i]
label = data["names"][i]
ax.text(x, y, z, label)
plt.legend(*sc.legend_elements(), bbox_to_anchor=(1.05, 1), loc=2)
plt.show()
Summary
This tutorial introduced affinity propagation and demonstrated how the technique determines clusters in financial market data. We applied affinity propagation to crypto market prices. In this way, we were able to identify 13 clusters of cryptocurrencies that have previously shown similar price behavior. The algorithm determined the number of clusters, so we did not need to specify them in advance. Finally, we plotted the cryptocurrencies and their clusters on 2D and 3D maps using a node distribution method and connected them on the map using covariance.
I hope this article helped show you an exciting way to visualize financial assets. If you have any questions or comments, please let me know.
Sources:
This article modifies some of the code from Scikit-learn and adapts it from the stock market to cryptocurrencies.
Very interesting !