What are Geographic Heat Maps?

Geographic heat maps are visual representations of data that use color or other visual encodings to show the density or intensity of data points in a geographic region. They are commonly used to represent data that is associated with a geographic location, such as population data, economic data, or weather data.

Geographic heat maps are typically created by overlaying a grid or mesh on a map and then assigning a color or other visual encoding to each grid cell based on the density or intensity of data points in that cell. The resulting heat map shows the distribution or pattern of data points across the geographic region and can provide valuable insights and information about the data.

Also: Color-Coded Cryptocurrency Price Charts in Python

What are Geographic Heat Maps used for?

What are the Potential Pitfalls of Using Geographic Heat Maps?

What is GeoPandas?

Creating Geographic Heat Maps with Python and GeoPandas

In this tutorial, we will learn how to create geographic heat maps using Python and the GeoPandas package. Geographic heat maps are visualizations that show the intensity of data at different locations on a map. They are commonly used to represent the distribution of a variable across a geographic area, and can be useful for identifying patterns, trends, and anomalies in the data. In this tutorial, we will learn how to create geographic heat maps using Python and the GeoPandas package. In the following, we will walk through the steps of loading, manipulating, and visualizing spatial data with GeoPandas, and demonstrate how to create geographic heat maps for covid-19.

The code is available on the GitHub repository.

Prerequisites

Before starting the coding part, make sure that you have set up your Python 3 environment and required packages. If you don’t have an environment set up yet, you can follow the steps in this tutorial to set up the Anaconda environment.

Also, make sure you install all required packages. We will be working with the following standard packages: 

• pandas
• NumPy
• math
• matplotlib

You can install packages using console commands:

pip install <package name>
conda install <package name> (if you are using the anaconda packet manager)

We will create geographic heat maps with the GeoPandas Python library. You can install GeoPandas via the console by using the following command:

• conda install –channel conda-forge geopandas
• pip install geopandas

Update (2020-09-23): With the release of Python 3.8, there is a new install procedure:

conda create -n geo_env
conda activate geo_env
conda config --env --add channels conda-forge
conda config --env --set channel_priority strict
conda install python=3 geopandas

Download the Geographic Map Data From Naturalearthdata

First, we will get the map with the geospatial data. Rendering maps with GeoPandas requires a shapefile. A shapefile is a DataFrame with some graphical data attached. For instance, some shapefiles show cities, countries, continents, or maps of the entire world. So in our case, the shapefile is a list of countries, whereby each country has its graphical representation in polygons. The example presented in this tutorial will use a world map.

Various sources on the web provide shapefiles for different geographical regions and in varying detail. For example, naturalearthdata.com provides a map of the world. To download the map, go to the natualearthdata webpage, and with a click on the green button, you can download version 4.1.0.

Once the download is complete, unpack the files into the folder of your Python notebook or a subfolder in the folder of your Python notebook (e.g., data/shapefiles/worldmap/).

Step #1 Loading the COVID-19 Data

Next, we retrieve the COVID-19 data for all countries via the statworx API. If you want to learn more about using REST APIs, check out this tutorial on accessing data sources via REST APIs.

Also: Accessing Remote Data Sources via REST APIs in Python

# Setting up Packages
import json
import country_converter as coco
from datetime import datetime, timedelta
import requests
import pandas as pd
import geopandas as gpd
import matplotlib.pyplot as plt
# Getting the data
PAYLOAD = {'code': 'ALL'}
URL = 'https://api.statworx.com/covid'
RESPONSE = requests.post(url=URL, data=json.dumps(PAYLOAD))
# Convert the response to a data frame
covid_df = pd.DataFrame.from_dict(json.loads(RESPONSE.text))
covid_df.head(3)

      date		day	month	year	cases	deaths	country		code	population	continent	cases_cum	deaths_cum
0	2019-12-31	31	12		2019	0		0		Afghanistan	AF		38041757.0	Asia		0			0
1	2020-01-01	1	1		2020	0		0		Afghanistan	AF		38041757.0	Asia		0			0
2	2020-01-02	2	1		2020	0		0		Afghanistan	AF		38041757.0	Asia		0			0

We continue by preparing the COVID-19 data for visualizing them on a heat map.

Step #2 Specifying a Shapefile

Next, we use the Geopandas library to read in a shapefile at “data/shapefiles/worldmap/ne_10m_admin_0_countries.shp”. We then select the columns “ADMIN,” “ADM0_A3”, and “geometry” from the shapefile and store them in a GeoDataFrame called “geo_df.” Finally, we display the first three rows of the GeoDataFrame.

# Setting the path to the shapefile
SHAPEFILE = 'data/shapefiles/worldmap/ne_10m_admin_0_countries.shp'
# Read shapefile using Geopandas
geo_df = gpd.read_file(SHAPEFILE)[['ADMIN', 'ADM0_A3', 'geometry']]
# Rename columns.
geo_df.columns = ['country', 'country_code', 'geometry']
geo_df.head(3)

	country		country_code	geometry
0	Indonesia	IDN				MULTIPOLYGON (((117.70361 4.16341, 117.70361 4...
1	Malaysia	MYS				MULTIPOLYGON (((117.70361 4.16341, 117.69711 4...
2	Chile		CHL				MULTIPOLYGON (((-69.51009 -17.50659, -69.50611

We have created a dataframe with three columns, as you can see above. The column geometry contains the graphical representation of countries. Now that we have prepared the data, we can plot our first geographic map. We create the map by using the GeoPandas plot function.

# Drop row for 'Antarctica'. It takes a lot of space in the map and is not of much use
geo_df = geo_df.drop(geo_df.loc[geo_df['country'] == 'Antarctica'].index)
# Print the map
geo_df.plot(figsize=(20, 20), edgecolor='white', linewidth=1, color='lightblue')

Step #3 Bringing It All Together

Next, we need to ensure that our data matches the country codes. The dataframe with the geospatial data of the world map contains country codes that adhere to iso3. However, our COVID-19 data uses iso2_codes. Luckily there is a country_converter available that does this job for us:

# Next, we need to ensure that our data matches with the country codes. 
iso3_codes = geo_df['country'].to_list()
# Convert to iso3_codes
iso2_codes_list = coco.convert(names=iso3_codes, to='ISO2', not_found='NULL')
# Add the list with iso2 codes to the dataframe
geo_df['iso2_code'] = iso2_codes_list
# There are some countries for which the converter could not find a country code. 
# We will drop these countries.
geo_df = geo_df.drop(geo_df.loc[geo_df['iso2_code'] == 'NULL'].index)

We have a list with all nations’ names (country) and codes (country_code). An additional column includes the geographical representation of each country.

Step #4 Preprocessing

Our COVID-19 data so far contains historical Covid-19 cases. We want to drop these historical cases and only get the data from the last day. Then we merge the data frames.

Before we plot the heat map, we have to specify a variable that determines the color of the countries on the map. Our goal is to color the countries depending on the growth rate of COVID-19 cases per day. The formula for the growth rate is ‘new cases’ / total present cases.

# We want to drop the history and only get the data from the last day
d = datetime.today()-timedelta(days=1)
date_yesterday = d.strftime("%Y-%m-%d")
# Preparing the data
covid_df = covid_df[covid_df['date'] == date_yesterday]
# Merge the two dataframes
merged_df = pd.merge(left=geo_df, right=covid_df, how='left', left_on='iso2_code', right_on='code')
# Delete some columns that we won't use
df = merged_df.drop(['day', 'month', 'year', 'country_y', 'code'], axis=1)
#Create the indicator values
df['case_growth_rate'] = round(df['cases']/df['cases_cum'], 2)
df['case_growth_rate'].fillna(0, inplace=True) 
df.head(3)

	country_x	country_code		geometry												iso2_code	date		cases	deaths	population	continent	cases_cum	deaths_cum	case_growth_rate
0	Indonesia	IDN					MULTIPOLYGON 	(((117.70361 4.16341, 117.70361 4...	ID			2020-06-28	1385.0	37.0	270625567.0	Asia		52812.0		2720.0		0.03
1	Malaysia	MYS					MULTIPOLYGON 	(((117.70361 4.16341, 117.69711 4...	MY			2020-06-28	10.0	0.0		31949789.0	Asia		8616.0		121.0		0.00
2	Chile		CHL					MULTIPOLYGON 	(((-69.51009 -17.50659, -69.50611...	CL			2020-06-28	4406.0	279.0	18952035.0	America		267766.0	5347.0		0.02

Step #5 Creating a Geographic Heat Map

In the previous step, we set up the data for our map. Next, we create the geographical heat map for the world.

We set the path to the shapefile and use Geopandas to read it. We then rename these columns. Next, we set the range for the choropleth and create a figure and axes for Matplotlib. We remove the axis and plot the choropleth using the data from the ‘df’ dataframe and the ‘case_growth_rate’ column, setting the edgecolor, linewidth, and cmap. We also add a title to the map and an annotation for the data source. Additionally, we create a colorbar as a legend, using the ScalarMappable function, and add it to the figure with a specified position.

# Print the map
# Set the range for the choropleth
title = 'Daily COVID-19 Growth Rates'
col = 'case_growth_rate'
source = 'Source: relataly.com \nGrowth Rate = New cases / All previous cases'
vmin = df[col].min()
vmax = df[col].max()
cmap = 'viridis'
# Create figure and axes for Matplotlib
fig, ax = plt.subplots(1, figsize=(20, 8))
# Remove the axis
ax.axis('off')
df.plot(column=col, ax=ax, edgecolor='0.8', linewidth=1, cmap=cmap)
# Add a title
ax.set_title(title, fontdict={'fontsize': '25', 'fontweight': '3'})
# Create an annotation for the data source
ax.annotate(source, xy=(0.1, .08), xycoords='figure fraction', horizontalalignment='left', 
            verticalalignment='bottom', fontsize=10)
            
# Create colorbar as a legend
sm = plt.cm.ScalarMappable(norm=plt.Normalize(vmin=vmin, vmax=vmax), cmap=cmap)
# Empty array for the data range
sm._A = []
# Add the colorbar to the figure
cbaxes = fig.add_axes([0.15, 0.25, 0.01, 0.4])
cbar = fig.colorbar(sm, cax=cbaxes)

As shown in the map above, countries in Central Asia and Africa currently report the highest COVID-19 growth rates.

There are different color palettes. You can use them by altering the cmap variable. Below is a sample of ready-to-use color scales. You can find more color scales on the matblotlib page.

Step #6 Zooming in on Specific Regions

We have observed that many African countries are currently reporting rising case numbers, so we create a new dataframe based on a filter for African countries using the list of country codes.

In the following, we create a geographic map specifically for Africa. We can zoom in on a continent or a country by filtering our dataframe. The code below will filter the spatial-geo data to African countries and plot the heat map. We plot the map for Africa using this new dataframe, setting the title of the map to ‘COVID-19 Growth Rate per Day in Africa’ and adding a source annotation to the bottom left corner of the map.

# The map shows that many african countries are currently reporting increasing case numbers
# Next we create a new df based on a filter for african countries
africa_country_list = ['ZM', 'BF', 'TZ', 'EG', 'UG', 'TN', 'TG', 'SZ', 'SD', 
                       'EH', 'SS', 'ZW', 'ZA', 'SO', 'SL', 'SC', 'SN', 'ST', 
                       'SH', 'RW', 'RE', 'GW', 'NG', 'NE', 'NA', 'MZ', 'MA', 
                       'MU', 'MR', 'ML', 'MW', 'MG', 'LY', 'LR', 'LS', 'KE', 
                       'CI', 'GN', 'GH', 'GM', 'GA', 'DJ', 'ER', 'ET', 'GQ', 
                       'BJ', 'CD', 'CG', 'YT', 'KM', 'TD', 'CF', 'CV', 'CM', 
                       'BI', 'BW', 'AO', 'DZ']
africa_map_df = df[df['iso2_code'].isin(africa_country_list)]
# Plot the map for Africa
title = 'COVID-19 Growth Rate per Day in Africa'
col = 'case_growth_rate'
source = 'Source: relataly.com \nGrowth Rate = New cases / All previous cases'
vmin = df[col].min()
vmax = df[col].max()
fig, ax = plt.subplots(1, figsize=(20, 9))
ax.axis('off')
africa_map_df.plot(column=col, ax=ax, edgecolor='0.8', linewidth=1, cmap=cmap)
ax.set_title(title, fontdict={'fontsize': '25', 'fontweight': '3'})
ax.annotate(source, xy=(0.24, .08), xycoords='figure fraction',
            horizontalalignment='left',
            verticalalignment='bottom', fontsize=10)
sm = plt.cm.ScalarMappable(norm=plt.Normalize(vmin=vmin, vmax=vmax), cmap=cmap)
cbaxes = fig.add_axes([0.35, 0.25, 0.01, 0.5])
{"type":"block","srcIndex":53,"srcClientId":"2ddd9666-6def-46e0-803e-4bf7b0366a27","srcRootClientId":""}cbar = fig.colorbar(sm, cax=cbaxes)

In case you encounter an error with the mapclassify-package, you can try the following command to reinstall it: conda install -c conda-forge mapclassify

Voilá, now we only see the African continent. The map shows that the countries in Africa that currently report the highest total case numbers are South Africa, Algeria, Morocco, Kamerun, and Egypt.

Let’s take a look at the total cases per country in Africa:

# Insert cases per population
# Alternative: africa_map_df2['cases_population'] = round(africa_map_df['cases_cum'] / africa_map_df['population'] * 100)
africa_map_df2 = africa_map_df.copy()
# Remove NAs
africa_map_df2.loc[: , 'cases_cum'].fillna(0, inplace=True)
# Show the data
africa_map_df2.head()
# Plot the map
title = 'Total COVID-19 Cases on the African Continent'
col = 'cases_cum'
source = 'Source: relataly.com '
vmin = africa_map_df2[col].min()
vmax = africa_map_df2[col].max()
fig, ax = plt.subplots(1, figsize=(20, 9))
ax.axis('off')
africa_map_df2.plot(column=col, ax=ax, edgecolor='1', linewidth=1, cmap=cmap)
ax.set_title(title, fontdict={'fontsize': '25', 'fontweight' : '3'})
ax.annotate(
    source, xy=(0.24, .08), xycoords='figure fraction', horizontalalignment='left', 
    verticalalignment='bottom', fontsize=10)
sm = plt.cm.ScalarMappable(norm=plt.Normalize(vmin=vmin, vmax=vmax), cmap=cmap)
cbaxes = fig.add_axes([0.35, 0.25, 0.01, 0.5])
cbar = fig.colorbar(sm, cax=cbaxes)

The highest growth rate was reported by South Sudan, followed by Botswana and Niger.

Step #7 Saving a Geo-Heat Maps to PNG

If you want to save the map, you can do this with the following command.

# Safe the map to a png
fig.savefig('map_export.png', dpi=300)

Summary

This article showed how to create geographic heat maps using the Geopandas library in Python. It showed how to read in a shapefile and create a choropleth map using the data from a dataframe. Additionally, the article explained how to filter the data to display maps of specific regions, in this case Africa. We showed how to prepare spatial data and color-code the maps using COVID-19 data. In addition, we filtered the DataFrame to create maps for specific regions, zoom in on specific areas, and alter the color style using different color maps.

Geographic heat maps can provide valuable insights into the distribution of data and help to identify patterns and trends. The technique of creating heat maps with Geopandas is a powerful tool for data visualization and can be applied to a wide range of geographical data.

I hope this article was helpful. If you have any questions or remarks, please write them in the comments.

Looking for more exciting map visualizations? Consider this relataly tutorial on predicting and visualizing crimes on a map of San Francisco.

Sources and Further Reading

https://geopandas.org/en/stable/getting_started.html

The post Geographic Heat Maps with GeoPandas: Visualizing COVID-19 Data in Python appeared first on relataly.com.

Correlation analysis is a powerful tool in financial market analysis, helping investors to better understand the interdependence of different assets. But what happens when an unprecedented global pandemic like COVID-19 shakes up the market? In this tutorial, we will show you how to create a correlation matrix in Python that will help you visualize the relationship between COVID-19 and various financial assets.

First, we will delve into the nitty-gritty of correlation coefficients and how to interpret them. We’ll focus specifically on the Pearson Correlation Coefficient, a popular measure used to quantify the strength of the relationship between two variables.

Next, we’ll dive right into the practical part of this tutorial and create a stock market correlation matrix in Python. Our matrix will measure the correlation between COVID-19 cases and various financial assets such as gold, Bitcoin, and other popular investments. With this matrix, investors can identify the extent to which COVID-19 has impacted different asset classes and make more informed investment decisions.

Whether you’re a seasoned investor or just starting out, this tutorial will equip you with the knowledge and tools you need to analyze the correlation between COVID-19 and financial markets. So, let’s dive in and start exploring the fascinating world of correlation analysis!

Also: Color-coded Cryptocurrency Price Charts with Python

Different Types of Correlation Coefficients

Pearson Correlation

Interpreting the Pearson Correlation Coefficient

Implementing a Correlation Matrix in Python

Prerequisites

Before starting the coding part, make sure that you have set up your Python 3 environment and required packages. If you don’t have an environment set up yet, you can follow this tutorial to set up the Anaconda environment. Also, make sure you install all required packages. In this tutorial, we will be working with the following standard packages: 

• pandas
• NumPy
• math
• matplotlib

In addition, we will be using the pandas-DataReader package and Seaborn for visualization. You can install packages using console commands:

pip install <package name>
conda install <package name> (if you are using the anaconda packet manager)

Step #1 Load Data

We begin by loading data about historic COVID-19 cases and price Information on different financial assets.

1.1 Load Historic COVID-19 Data

We begin by downloading the COVID-19 data. For this purpose, we will use the Statworx API. It provides historical time series data on the number of COVID-19 cases in different countries. In addition, the data contains the number of casualties. If you are not yet familiar with APIs, consider my recent tutorial on working with APIs in Python.

# A tutorial for this file is available at www.relataly.com

# Imports
import pandas as pd
import pandas_datareader as web
import numpy as np
from datetime import datetime
import seaborn as sns
from matplotlib import pyplot as plt
import matplotlib.dates as mdates
import requests
import json
from pandas.plotting import register_matplotlib_converters

# Load second Dataset with Corona Cases
payload = {"code": "ALL"}
URL = "https://api.statworx.com/covid"
response = requests.post(url=URL, data=json.dumps(payload))
df_covid = pd.DataFrame.from_dict(json.loads(response.text))
# df_covid = df_covid[df_covid['code'] == 'US']

# Add the date column as variable
df_covid["Date"] = pd.to_datetime(df_covid["date"])

# Delete some columns that we won't use
df_covid.drop(
    ["day", "month", "year", "country", "code", "population", "date"],
    axis=1,
    inplace=True,
)

# Summarize cases over all countries
df_covid = df_covid.groupby(["Date"]).sum()
df_covid.head()

			cases	deaths	cases_cum	deaths_cum
Date				
2019-12-31	27		0		27			0
2020-01-01	0		0		27			0
2020-01-02	0		0		27			0
2020-01-03	17		0		44			0
2020-01-04	0		0		44			0

1.2 Loading Data on Selected Financial Assets

We continue by downloading historical price data on different financial assets. For this purpose, we use the Yahoo Finance API. We limit the period to the time after the first documented COVID-19 cases. When you execute the code of this tutorial as it is, you will receive price information for the following financial assets:

Be aware that stock symbols can change from time to time. If the API does not find a specific stock symbol, you have to look up the current Symbol on Yahoo Finance.

df_covid_new = df_covid.copy()

# Read the data for different assets
today_date = datetime.today().strftime("%Y-%m-%d")
start_date = "2020-01-01"
asset_dict = {
    "^GSPC": "SP500",
    "DAX": "DAX",
    "^N225": "N225",
    "ES=F": "SP500FutJune20",
    "LHA.DE": "Lufthansa",
    "AAL": "AmericanAirlines",
    "NFLX": "Netflix",
    "AMZN": "Amazon",
    "AAPL": "Apple",
    "MSFT": "Microsoft",
    "GOOG": "Google",
    "BTC-USD": "BTCUSD",
    "ETH-USD": "ETHUSD",
    "CL=F": "Oil",
    "GC=F": "Gold",
    #"SM=F": "Soybean",
    "ZB=F": "UsTreasuryBond",
    "GBPEUR=X": "GBPEUR",
    "EURUSD=X": "EURUSD",
    "CHFEUR=X": "CHFEUR",
    "GBPUSD=X": "GBPUSD"}

col_list = []
# Join the dataframes
for key, value in asset_dict.items():
    print(key, value)    
    try:
        df_temp = web.DataReader(
            key, start=start_date, end=today_date, data_source="yahoo")
    except ValueError: 
        print(f' {key} symbol not found')
    # convert index to Date Format
    df_temp.index = pd.to_datetime(df_temp.index) 
    df_temp.rename(columns={"Close": value}, inplace=True) # Rename Close Column       
    df_covid_new = pd.merge(
        left=df_covid_new,
        right=df_temp[value],
        how="inner",
        left_index=True, right_index=True)     

df_covid_new.head()

	cases	deaths	cases_cum	deaths_cum	SP500	DAX			N225		SP500FutJune20	Lufthansa		AmericanAirlines	...	Google		BTCUSD		ETHUSD		Oil	Gold	UsTreasuryBond	GBPEUR	EURUSD	CHFEUR	GBPUSD
Date																					
2020-01-06	0		0			59			0		3246.280029	28.004999	23204.859375	3243.50	15.340	27.320000			...	1394.209961	7769.219238	144.304153	63.270000	1566.199951		157.84375	1.17169	1.116196	0.922110	1.308010
2020-01-07	0		0			59			0		3237.179932	27.955000	23575.720703	3235.25	15.365	27.219999			...	1393.339966	8163.692383	143.543991	62.700001	1571.800049		157.40625	1.17635	1.119799	0.922212	1.317003
2020-01-08	0		0			59			0		3253.050049	28.260000	23204.759766	3260.25	15.540	27.840000			...	1404.319946	8079.862793	141.258133	59.610001	1557.400024		156.37500	1.17551	1.115474	0.925181	1.311372
2020-01-09	0		0			59			0		3274.699951	28.450001	23739.869141	3276.00	16.160	27.950001			...	1419.829956	7879.071289	138.979202	59.560001	1551.699951		156.81250	1.17912	1.111321	0.924505	1.310513
2020-01-10	0		0			59			0		3265.350098	28.500000	23850.570312	3264.75	15.815	27.320000			...	1429.729980	8166.554199	143.963776	59.040001	1557.500000		157.62500	1.17620	1.111111	0.924796	1.307019
5 rows × 24 columns

You can add assets of your choice to the asset list if you want. You can find the respective symbols on finance.yahoo.com.

Step #2 Exploring the Data

Next, we will visualize the historical data using line charts.

# Create lineplots
list_length = df_covid_new.shape[1]
ncols = 6
nrows = int(round(list_length / ncols, 0))
height = list_length/3 if list_length > 30 else 16

fig, axs = plt.subplots(nrows=nrows, ncols=ncols, sharex=True, figsize=(20, height))

for i, ax in enumerate(fig.axes):
        if i < list_length:
            sns.lineplot(data=df_covid_new, x=df_covid_new.index, y=df_covid_new.iloc[:, i], ax=ax)
            ax.set_title(df_covid_new.columns[i])
            ax.tick_params(labelrotation=45)

plt.show()

We can easily spot pairs that seem to have experienced similar price developments. This does not mean, however, that these pairs are correlated.

Step #3 Correlation Matrix

Next, we will calculate the correlation matrix. Various Python libraries make this an easy task that only requires a few lines of code. We will use the standard math package for this purpose.

# Plotting a diagonal correlation matrix
sns.set(style="white")

# Compute the correlation matrix
df = pd.DataFrame(df_covid_new, columns=col_list)
corr = df_covid_new.corr()
corr

					cases	deaths	cases_cum	deaths_cum	SP500	DAX	N225	SP500FutJune20	Lufthansa	AmericanAirlines	...	Google	BTCUSD	ETHUSD	Oil	Gold	UsTreasuryBond	GBPEUR	EURUSD	CHFEUR	GBPUSD
cases				1.000000	0.853512	0.972691	0.966481	0.663638	0.519676	0.660547	0.659832	-0.451801	-0.413463	...	0.796671	0.898456	0.899876	0.073393	0.719520	0.147347	-0.566227	0.843788	-0.538949	0.513913
deaths				0.853512	1.000000	0.778833	0.804270	0.399756	0.259080	0.400126	0.395697	-0.590251	-0.589090	...	0.567708	0.705201	0.718329	-0.228573	0.664476	0.399694	-0.574079	0.628463	-0.291254	0.245614
cases_cum			0.972691	0.778833	1.000000	0.974553	0.714616	0.571317	0.711905	0.711552	-0.379420	-0.325739	...	0.812816	0.922179	0.932026	0.142586	0.682001	0.059693	-0.516654	0.865846	-0.584541	0.584691
deaths_cum			0.966481	0.804270	0.974553	1.000000	0.712595	0.587606	0.681964	0.709312	-0.498761	-0.441631	...	0.808086	0.875724	0.925765	0.097746	0.805602	0.193165	-0.626253	0.902159	-0.603867	0.529622
SP500				0.663638	0.399756	0.714616	0.712595	1.000000	0.960100	0.956142	0.999766	0.140961	0.205127	...	0.944084	0.806056	0.801970	0.623960	0.553991	-0.359058	-0.043646	0.738902	-0.791377	0.853893
DAX					0.519676	0.259080	0.571317	0.587606	0.960100	1.000000	0.934535	0.960816	0.246646	0.304234	...	0.860881	0.678125	0.688038	0.715992	0.500840	-0.387279	-0.002362	0.685518	-0.844509	0.826270
N225				0.660547	0.400126	0.711905	0.681964	0.956142	0.934535	1.000000	0.956710	0.240638	0.281306	...	0.922091	0.829050	0.761729	0.655562	0.425364	-0.436453	-0.005655	0.673853	-0.790071	0.810057
SP500FutJune20		0.659832	0.395697	0.711552	0.709312	0.999766	0.960816	0.956710	1.000000	0.147155	0.211133	...	0.943475	0.804529	0.799886	0.627447	0.549565	-0.363198	-0.039701	0.736997	-0.792258	0.855152
Lufthansa			-0.451801	-0.590251	-0.379420	-0.498761	0.140961	0.246646	0.240638	0.147155	1.000000	0.964624	...	-0.006089	-0.135931	-0.296115	0.629831	-0.665533	-0.853762	0.815127	-0.388975	-0.107357	0.262015
AmericanAirlines	-0.413463	-0.589090	-0.325739	-0.441631	0.205127	0.304234	0.281306	0.211133	0.964624	1.000000	...	0.026610	-0.115151	-0.245080	0.658176	-0.603162	-0.877327	0.790366	-0.312451	-0.143469	0.330665
Netflix				0.750950	0.701806	0.721492	0.840104	0.601819	0.523924	0.493603	0.596449	-0.637187	-0.578967	...	0.672056	0.614683	0.749042	-0.027917	0.914606	0.438247	-0.652950	0.766065	-0.460004	0.338608
Amazon				0.801935	0.710040	0.776041	0.887487	0.669833	0.597223	0.564001	0.665996	-0.591990	-0.528531	...	0.732905	0.672651	0.809639	0.049571	0.936733	0.365580	-0.664869	0.848771	-0.562987	0.428907
Apple				0.840178	0.631516	0.862322	0.917166	0.843786	0.765495	0.750124	0.841533	-0.357089	-0.275023	...	0.860493	0.800416	0.906042	0.295665	0.851025	0.081060	-0.499164	0.927081	-0.719334	0.673724
Microsoft			0.772067	0.647593	0.751721	0.849898	0.792196	0.723458	0.689305	0.788468	-0.416892	-0.354098	...	0.833358	0.723236	0.819949	0.206249	0.871319	0.209342	-0.496853	0.807662	-0.598330	0.529434
Google				0.796671	0.567708	0.812816	0.808086	0.944084	0.860881	0.922091	0.943475	-0.006089	0.026610	...	1.000000	0.902355	0.866670	0.492750	0.593879	-0.219884	-0.174525	0.765421	-0.713271	0.770065
BTCUSD				0.898456	0.705201	0.922179	0.875724	0.806056	0.678125	0.829050	0.804529	-0.135931	-0.115151	...	0.902355	1.000000	0.942019	0.315591	0.568836	-0.099474	-0.285379	0.777073	-0.620303	0.685506
ETHUSD				0.899876	0.718329	0.932026	0.925765	0.801970	0.688038	0.761729	0.799886	-0.296115	-0.245080	...	0.866670	0.942019	1.000000	0.242502	0.740186	0.068097	-0.419289	0.886153	-0.644605	0.696074
Oil					0.073393	-0.228573	0.142586	0.097746	0.623960	0.715992	0.655562	0.627447	0.629831	0.658176	...	0.492750	0.315591	0.242502	1.000000	-0.035808	-0.685471	0.344647	0.261168	-0.615400	0.626496
Gold				0.719520	0.664476	0.682001	0.805602	0.553991	0.500840	0.425364	0.549565	-0.665533	-0.603162	...	0.593879	0.568836	0.740186	-0.035808	1.000000	0.485554	-0.672429	0.815864	-0.489188	0.381673
UsTreasuryBond		0.147347	0.399694	0.059693	0.193165	-0.359058	-0.387279	-0.436453	-0.363198	-0.853762	-0.877327	...	-0.219884	-0.099474	0.068097	-0.685471	0.485554	1.000000	-0.667468	0.154001	0.278546	-0.412731
GBPEUR				-0.566227	-0.574079	-0.516654	-0.626253	-0.043646	-0.002362	-0.005655	-0.039701	0.815127	0.790366	...	-0.174525	-0.285379	-0.419289	0.344647	-0.672429	-0.667468	1.000000	-0.586152	0.230223	0.187170
EURUSD				0.843788	0.628463	0.865846	0.902159	0.738902	0.685518	0.673853	0.736997	-0.388975	-0.312451	...	0.765421	0.777073	0.886153	0.261168	0.815864	0.154001	-0.586152	1.000000	-0.756216	0.686032
CHFEUR				-0.538949	-0.291254	-0.584541	-0.603867	-0.791377	-0.844509	-0.790071	-0.792258	-0.107357	-0.143469	...	-0.713271	-0.620303	-0.644605	-0.615400	-0.489188	0.278546	0.230223	-0.756216	1.000000	-0.711504
GBPUSD				0.513913	0.245614	0.584691	0.529622	0.853893	0.826270	0.810057	0.855152	0.262015	0.330665	...	0.770065	0.685506	0.696074	0.626496	0.381673	-0.412731	0.187170	0.686032	-0.711504	1.000000
24 rows × 24 columns

The matrix shows the Pearson correlation coefficients of all the pairs (X, Y) in our dataset.

Step #4 Visualizing the Correlation Matrix in a Heatmap

Heatmaps are an excellent choice for visualizing a correlation matrix. The heatmap applies a color palette to represent numeric values on a scale in different colors. This makes it easier to capture differences and similarities among the correlation coefficients. In Python, we can create heatmaps using the Seaborn package.

# Generate a mask for the upper triangle
mask = np.triu(np.ones_like(corr, dtype=np.bool))

# Set up the matplotlib figure
f, ax = plt.subplots(figsize=(11, 9))

# Generate a custom diverging colormap
cmap = "RdBu"

# Draw the heatmap with the mask and correct aspect ratio
sns.heatmap(
    corr,
    mask=mask,
    cmap=cmap,
    center=0,
    square=True,
    linewidths=0.5,
    cbar_kws={"shrink": 0.5},
)

The correlation matrix is symmetric. This is because the correlation between a pair of variables X and Y is the same as between Y and X.

Step #5 Interpretation

The heatmap uses a color palette that ranges from blue (positive correlation) over white (no correlation) to red (negative correlation). The different shades of the three colors visualize the extent of the correlation. We can distinguish between correlated pairs, uncorrelated pairs, and negatively correlated pairs. We will compare the different asset classes step by step in the following.

5.1 Stock Market Indices / COVID-19

Let us start with the pairs of Stock market indices and COVID-19 data. The heatmap signals a negative correlation between the indices (DAX, S&P500, NIKI) and COVID-19. In other words, when the number of cases rises, stock market indices tend to fall in value. If we look precisely, the total number of new cases seems more correlated than the number of cases (cases_cum) or deaths (deaths_cum). In addition, one can observe that the stock market indices are correlated.

5.2 Stock Market Indices / Online Service Provider Stocks

The situation is heterogeneous when we compare the stock markets with the shares of online service providers. There is a positive correlation between the shares of Microsoft and Google and the overall development of the markets. On the other hand, the shares of Netflix, Amazon, and Apple are hardly correlated with market development.

5.3 Stock Market Indices / Airline Stocks

Airlines are heavily affected by the pandemic. Thus it is plausible that we observe a strong positive correlation between airline stocks and the general stock market indices.

5.4 Stock Market Indices / Crypto-Currencies

Next, we compare Cryptocurrencies with the stock market indices. The results are surprising. BTC-USD correlates surprisingly strong positive with the general development of the stock markets. However, the correlation is only slightly positive for ETH-USD and the markets.

5.5 COVID-19 / Currency Exchange Rates

The correlation between exchange rates and COVID-19 cases is relatively weak. Only GBP/EUR, EUR/USD, and GBP/USD show a slightly negative correlation. An exception is CHF/EUR, which positively correlates to the number of COVID-19 cases.

5.6 Treasury Bonds / Resources

Looking at the coefficients of resources and US Treasury Bonds, we can observe a strong negative correlation between COVID-19 cases and the oil price and a strong positive correlation with the gold price.

5.7 Crypto-Currencies / Resources

Finally, let us consider the coefficients of resources and cryptocurrencies. It is noticeable that BTCUSD correlates with the oil price. Based on the absence of a correlation with gold, one might conclude that BTC-USD is not a comparable crisis currency. However, the correlation between market indices and cryptocurrencies such as ETH-USD is relatively low. Thus, they were less affected by the recent market slump.

Also: Stock Market Prediction using Multivariate Data

Summary

Congratulation, you have reached the end of this tutorial! In this article, we have load data on COVID-19 and financial assets via an API. We have created a correlation matrix in Python that shows the linear correlation between financial assets and COVID-19 cases. Finally, we have visualized the matrix in a heatmap and concluded the correlation of different asset pairs. However, we must remember that we may still be unaware of potential non-linear correlations.

Please show your appreciation by leaving a like or comment if you found this article helpful.

And if you are interested to learn more about an advanced use case for correlation analysis, please take a look at this article on clustering cryptocurrencies.

Sources and Further Reading

• YouTube tutorial that explains the math behind the correlation
• Andriy Burkov (2020) Machine Learning Engineering
• Uncover Hidden Patterns in Financial Markets using Affinity Propagation Clustering in Python

Images created with Midjourney.

The post Correlation Matrix in Python: How Correlated are COVID-19 Cases and Different Financial Assets? appeared first on relataly.com.

REST APIs provide straightforward access to remote data sources. Data scientists should learn about REST APIs because APIs (Application Programming Interfaces) are an important way for data scientists to access data from other sources. By using REST APIs, data scientists can access data from a wide range of sources, including databases, web services, and other applications. This allows data scientists to easily integrate data from different sources and use it in their analysis and modeling. Additionally, knowing how to use REST APIs can also be useful for data scientists who want to share their own data or models with others by building APIs that can be accessed by other applications. This article briefly describes how to access remote data sources via REST APIs in Python.

The article proceeds as follows: We begin with a brief introduction to RESTful APIs. Then, we will look at two examples of how to access remote APIs. We will define API requests, learn about parameters, and request data from an API. We will handle the response and save it to a dataframe or a local CSV file.

Also: Streaming Tweets and Images via the Twitter API in Python

What is an API?

In Data Science, REST APIs are often used as a modern shortcut to consume remote data sources and services on the internet or make them available to others. Whether these sources are publicly available or reside in corporate networks, data scientists need to understand how to work with API. In the broader sense, an API is a contract between the provider and the consumer of a web service, who communicate and exchange data. The agreement is necessary because communication can lead to misunderstandings, for example, when one party sends information that is not as expected. The contract avoids such misunderstandings by defining standards for what the parties communicate and how it is communicated.

What are RESTful APIs?

A popular architectural style used to design APIs is the Representational State Transfer (REST) standard. REST has become very popular in recent years and is often considered a more straightforward and modern alternative to the traditional Simple Object Access Protocol (SOAP). Both SOAP and REST are based on established rules, compliance with which is the basis of automated information exchange. However, in data science, REST is now the most common.

If you work with public APIs in Python, you will most likely use the REST protocol. A service provider that operates a REST API exposes an URL that receives requests. Requests to the resource URL can have a JSON, HTML, or XML format payload. A REST API will typically return the response in JSON format, but other formats, such as comma-separated values (CSV), are also possible. REST defines different HTTP methods (GET, HEAD, POST, PUT, PATCH, DELETE, CONNECT, OPTIONS and TRACE). However, the most important ones are:

• GET: Used to request data from a data provider.
• POST: Typically used to send new data to the service provider. Sometimes also used to define the data that the APIs returns in response to subsequent requests.
• PUT: Used to update data at the service provider.

Next, let’s look at how we can interact with RET APIs in Python.

How can we interact with REST APIs in Python?

There are several ways to access REST APIs in Python, including:

1. Using the requests library: The requests library is a popular Python library for making HTTP requests. It provides a simple, intuitive API for sending and receiving requests, and supports a wide range of HTTP methods, such as GET, POST, and DELETE.
2. Using the urllib library: The urllib library is a built-in Python library for working with URLs and HTTP requests. It provides a lower-level API than the requests library but is still relatively easy to use and allows you to customize your requests in more detail.
3. Using the HTTP.client library: The HTTP.client library is another built-in Python library for working with HTTP requests. It provides a more comprehensive and robust set of tools for working with HTTP requests but may require more programming effort to use than the other options.
4. Service-specific libraries: Many popular online services can be accessed using Python libraries that offer built-in functionality for interacting with REST-based APIs. For example, Twitter, Reddit, or Yahoo Finance offer libraries that make interacting with their REST APIs much easier.

The choice of which Python library to use for accessing REST APIs will depend on your specific needs and preferences. You may want to try out a few different options and see which one works best for your project. In the following tutorial, we will work with the requests library.

Requesting historical COVID-19 Data from the Statworx API with the Request Library

There are many APIs out there that provide more or less reliable data on COVID-19 cases. A good one to use is api.statworx.com/covid. This API offers historical data on the number of COVID-19 cases. The API is very accessible since it does not require an authentication key.

In the following, we will go through two different cases on how to request data from REST APIs in Python. First, we use the Requests library to request COVID-19 data from the statworx library. We will call this API using the request package, a standard package to interact with APIs in Python. An alternative would be to use a Python package that provides API-specific functions to interact with the API. We will look at this case in the second example.

The code is available on the GitHub repository.

Prerequisites

Before starting the coding part, ensure you have set up your Python 3 environment and required libraries. If you don’t have an environment, follow the steps in this tutorial to set up the Anaconda environment.

Also, make sure you install all required packages. In this tutorial, we will be working with the following standard packages: 

• pandas
• NumPy
• math
• matplotlib

You can install packages using console commands:

• pip install <package name>
• conda install <package name> (if you are using the anaconda packet manager)

In addition, we will be working with the Requests library. The requests library is a standard HTTP library that interacts with REST APIs. It provides functionality to send an HTTP request to an API and receive a response. Since the requests library is a native Python, you usually won’t need to install it.

Step #1 Define the Payload

In order to make a request to an API, it is necessary to specify the URL that the request should be sent to. This URL is typically provided by the API documentation and may include certain parameters or queries to specify the specific data that you want to retrieve. For example, you might want to specify a specific date range or filter the results by a certain category.

In the following, we send a post request to the statworx.com API and get back COVID-19 data in JSON format as a response. We can then convert the response into a dataframe. With the following code, we can send an HTTP request to the URL provided and will get the requested data back in JSON format:

# the data is provided from the european centre for disease prevention and control
from datetime import date
import requests
import json
import pandas as pd
# import matplotlib.pyplot as plt

# define the payload that will be sent to the api endpoint, and the endpoint url
# code defines the countries for which we will retrieve data
# to retrieve data for specific countries use e.g. {'country': 'Germany'}
payload = {"code": "ALL"}  
URL = "https://api.statworx.com/covid"

We can change the data to get back in the response by altering the parameters in our request. For example, we have specified the country code as “US” in the payload. Thus, the response contains only COVID-19 data for the US. If we want data for all countries, we need to set the code to ALL.

Step #2 Call the REST API Endpoint

Once you have defined the URL, you can use Python’s built-in “requests” library to send an HTTP request to the API and retrieve the data. This can be done using the “get” method, which sends a GET request to the specified URL and returns a response object containing the data returned by the API.

It is also important to consider any authentication or authorization requirements that the API may have. The statworx API does not require authentication. However, some APIs may require you to provide a valid API key or other credentials in order to access the data. These requirements will also be specified in the API documentation.

# call the api
response = requests.post(url=URL, data=json.dumps(payload))
response # if the request was successful, you should see a response code 200

<Response [200]>

Step #3 Convert the Data to a DataFrame

When making an API request using Python, the API will typically return a response in JSON format. JSON (JavaScript Object Notation) is a lightweight data interchange format that is easy for humans to read and write and easy for machines to parse and generate. It is often used as a default data format in REST APIs because it is easy to work with and can be easily converted into other data formats such as CSV or Pandas DataFrames.

A common way to handle the response from an API is to convert it into a DataFrame using the “JSON” method of the Pandas library. This method takes a JSON object or a list of JSON objects and converts it into a DataFrame. Once the data is in a DataFrame, it can be easily processed and analyzed using the various tools and functions provided by Pandas.

It is also possible to directly parse the JSON response using Python’s built-in “JSON” module. This can be useful if you want to extract specific pieces of data from the response or if you want to perform additional processing on the data before converting it into a DataFrame.

# convert the response data to a data frame
df = pd.DataFrame.from_dict(json.loads(response.text))
df

	date		day	month	year	cases	deaths	country		code	population	continent	cases_cum	deaths_cum
0	2019-12-31	31	12		2019	0		0		Afghanistan	AF		38041757.0	Asia		0			0
1	2020-01-01	01	01		2020	0		0		Afghanistan	AF		38041757.0	Asia		0			0
2	2020-01-02	02	01		2020	0		0		Afghanistan	AF		38041757.0	Asia		0			0
3	2020-01-03	03	01		2020	0		0		Afghanistan	AF		38041757.0	Asia		0			0
4	2020-01-04	04	01		2020	0		0		Afghanistan	AF		38041757.0	Asia		0			0
...	...			...	...		...		...		...		...			...		...			...			...			...

Step #4 Filter the Data

Now let’s make something out of the data and create a simple plot. Our goal is to create a lineplot that shows how cases have developed for different countries. As part of the data preprocessing, we will reduce the data to include a smaller number of selected countries.

# convert date column to date format
df.loc[:, "date"] = pd.to_datetime(df["date"], format="%Y-%m-%d")

# filter specific countries
list_of_countries = ["Germany", "Switzerland", "France", "Spain", "Canada"]
df_1 = df[df["country"].isin(list_of_countries)]

# filter the data to a specific timeframe
df_new = df_1[df_1["date"] > "2020-01-15"]
df_new.head()

		date		day	month	year	cases	deaths	country	code	population	continent	cases_cum	deaths_cum
10332	2020-01-16	16	01		2020	0		0		Canada	CA		37411038.0	America		0			0
10333	2020-01-17	17	01		2020	0		0		Canada	CA		37411038.0	America		0			0
10334	2020-01-18	18	01		2020	0		0		Canada	CA		37411038.0	America		0			0
10335	2020-01-19	19	01		2020	0		0		Canada	CA		37411038.0	America		0			0
10336	2020-01-20	20	01		2020	0		0		Canada	CA		37411038.0	America		0			0

Step #5 Plot the Case Counts

Finally, we use Matplotlib to create a lineplot that visualizes the data that we have received from the API.

# create separate plot lines for each country in the dataset
fig, ax1 = plt.subplots(figsize=(12, 8))
plt.ylabel("Total Cases", fontsize=20, color="black")
for countryname in list_of_countries:
    x = df_new[df_new["country"] == countryname]["date"]
    y = df_new[df_new["country"] == countryname]["cases_cum"]
    plt.plot(x, y, label=countryname)

plt.legend(list_of_countries, loc="upper left")
plt.show()

If your chart looks similar to the one above, you can be confident that you have successfully loaded the data into our project.

Summary

In conclusion, this tutorial has demonstrated how to use REST APIs in Python to access and retrieve data from remote sources. We used the statworx API to fetch COVID-19 data and used the Pandas library to convert the data into a DataFrame. We also created a small sample plot to visualize the number of COVID-19 cases in different countries.

By following the steps outlined in this tutorial, you should now be able to use Python to access and work with data from a variety of sources using REST APIs. This can be a powerful tool for data scientists and analysts, allowing them to easily retrieve and analyze data from a wide range of sources in order to gain insights and make informed decisions.

I hope this post was helpful. If you have any remarks or questions, let me know.

Sources and Further Reading

1. statworx API
2. https://pypi.org/project/requests/

Relataly API Tutorials

• 

  Generating Detailed Images with OpenAI DALL-E and ChatGPT in Python: A Step-By-Step API Tutorial

• 

  Unleashing the Power of ChatGPT and Other OpenAI GPT Language Models in Python A Guide to Using APIs

• 

  Univariate Stock Market Forecasting using Facebook Prophet in Python

• 

  On-Chain Analytics: Metrics for Analyzing Blockchains in Python

• 

  Requesting Crypto Price Data from the Gate.io REST API in Python

• 

  Posting Tweets On Twitter using Python and Tweepy

• 

  Requesting Crypto Prices from the Coinmarketcap API using Python

The post Accessing Remote Data Sources via REST APIs in Python appeared first on relataly.com.