Streaming Bitcoin Prices in Real-time via the Coinmarketcap API using Python

0

In this tutorial, you will learn how to stream bitcoin price quotes in real-time via a public API and save the data to a local SQLite database. Streaming price data lays the foundation for many exciting machine learning applications in finance, such as generating trading signals for buy and sell orders. Such use cases work on real-time price data that is typically obtained from a REST API endpoint. In this example, we will use Python to query the API from Coinmarketcap.com.

The rest of this article is structured as follows: Before we can get our hands dirty in the coding part, you first need to obtain a free API key from Coinmarketcap. We will then use this key when we make calls to stream price quotes. To store the retrieved data to a local SQLite database, we use Peewee, which is a lightweight SQLite framework. Finally, I show how you can query the obtained price quotes from our local database and use them later on, for example, to train a price prediction model.

Acquiring an API Key from Coinmarketcap

Coinmarketcap is one of the most popular websites in the crypto space. It provides price data, charts, and other information for a wide range of cryptocurrencies such as Bitcoin and Ethereum. Sending requests to the Coinmarketcap API requires an API key. In order to acquire your personal API key, you need to register an account. During the registration process, you will have the choice between different pricing plans. Fortunately, there is a free plan for personal use. It is a bit limited in terms of the number of API requests, but to get familiar with the API, is more than sufficient.

coinmarketcap api pricing plans

After login into your account, you can display the API Key under the “Overview” section. For the sample code to work, you need to create a YAML file in the same folder where your Python project files are located named coinmarketcap.yml. Insert your code into this file as follow:

key: "your coinmarketcap api key"

I’ll later show how you can import and access this key from your python code.

Each call to the Coinmarketcap API consumes credits. The number of calls and data we can retrieve via API is limited to a few 100 credits for the free plan. On the overview page, you can see how many of your daily and monthly credits are still available.

coinmarketcap api key
Overview page of the coinmarketcap API

Python Example: Streaming Quotes from the Coinmarketcap API

In the following, we implement a regular call to the Coinmarketcap API. To do this, we create a loop that sends regular calls to the REST endpoint and returns price data in JSON format. After each call to the API, we create an object for the price point and store it in a local SQLite database using the Peewee framework.

Prerequisites

Before we begin with the coding part, please make sure that you have setup your Python 3 environment and required packages. If you don’t have a Python environment available yet, you can follow this tutorial to setup the Anaconda Python environment.

Also make sure you install required packages. In this tutorial, we will be working with the following standard packages: 

In addition, for visualization purposes, we will use seaborn. To interact with the SQLite DB we will use peewee, which is a lightweight Object-Relational-Mapper (ORM) framework that lets us interact with SQLite in an object-oriented and more convenient way.

You can install packages using console commands:

  • pip install <package name>
  • conda install <package name> (if you are using the anaconda packet manager)

Step #1 Create a Relational Model in SQLite

We begin by defining a relational data model in SQlite, in which we will store the obtained price quotes. For this purpose, we will use Peewee, which is an object-oriented framework that allows us to create objects and persist them to the SQLite database. Our model contains two tables:

  • Run: A table in which we will store an ID and a timestamp, that makes it easier to identify past runs. Everything we start querying data from the API, we will add a new runid to this table.
  • The second table is called Price and contains the price quotes for a predetermined interval. We will store the price quotes in this table, and reference the runid via a foreign key.
from peewee import *
from datetime import date, timedelta, datetime
import numpy as np 
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import time
import json
import yaml
from requests import Request, Session
from requests.exceptions import ConnectionError, Timeout, TooManyRedirects

# persist information
db = SqliteDatabase('assets.db')

class BaseModel(Model):
    class Meta:
        database = db

class Run(BaseModel): # This model uses the "assets.db" database.
    timestamp = DateTimeField(unique=True)
    symbol = CharField()
    id = AutoField()

class Price(BaseModel): # This model uses the "assets.db" database.
    datetime = DateTimeField(unique=True)
    volume_24 = FloatField()
    price = FloatField()
    runid = ForeignKeyField(Run, backref='prices')
         
# By default, Peewee includes an IF NOT EXISTS clause when creating tables. 
def create_tables():
    with db:
        db.create_tables([Price])
        db.create_tables([Run])

def drop_tables():
    with db:
        db.drop_tables([Price])
        db.drop_tables([Run])
        
create_tables()

#drop_tables()    

If you want to drop the tables again, you can call our custom drop_tables function.

Step #2 Streaming Price Quotes

Once, we have created the relational data model, we can define a request to the Coinmarketcap API and send a streaming request. Coinmarketcap offers different APIs. The one we will be using is cryptocurrency/listings/latest. This API provides the latest price data for a number of cryptocurrencies.

Before we query the API, we need to define some basic parameters such as the conversion currency and the “limit”. We will select USD as convert and set the limit to 1. This value will lead to the API only returning the price of Bitcoin. A value of 2 will additionally also return the price of Ethereum (ETH). Higher values will give back additional cryptocurrencies depending on their marketcap ranking.

I have set the query interval to 10 seconds, but you can change it to any other value. However, be aware that you will deplete your free credits quickly if you request data by the second. I have limited the number of price quotes retrieved to 100. If you want to retrieve price quotes for longer periods, increase the timeframe.

Once we have defined these settings, we can query the API.

symbol='BTCUSD'
query_interval = 10 # in seconds
query_number = 100 # the number of queries after which the API stops

# load the API key from our local file
with open(r'C:/Users/Flo/relataly-public-python-tutorials/coinmarketcap.yml') as file:
    apikey = yaml.load(file, Loader=yaml.FullLoader)['key']

# Define some essential API parameters
# Coinmarketcap API for latest market ticker quotes and averages for cryptocurrencies and exchanges.
# https://coinmarketcap.com/api/documentation/v1/#operation/getV1CryptocurrencyListingsLatest
url = 'https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest'
parameters = {
  'start':'1',
  'limit':'1',
  'convert':'USD'
}
headers = {
  'Accepts': 'application/json',
  'X-CMC_PRO_API_KEY': apikey,
}

session = Session()
session.headers.update(headers)

# create a new run and save it to our local SQLite DB
run_timestamp = datetime.now()
run = Run(symbol=symbol, timestamp = run_timestamp)
run.save()
current_run_id = run.id 
print(f'run_id: {current_run_id} - timestamp: {run_timestamp} - interval: {query_interval} - number of queries: {query_number}')

# query the coinmarketcap API every x seconds
for s in range(0, query_number):
    try:
        response = session.get(url, params=parameters)
        data = json.loads(response.text)
        #print(data)
        
        # response - quote
        data_quote = data['data'][0]['quote']['USD']
        price = np.round(data_quote['price'], 1) # the price
        volume_24 = np.round(data_quote['volume_24h'], 1) # the volume in the last 24 hours
        
        # response - status
        data_status = data['status']
        api_timestamp = data_status['timestamp'] # the timestamp of the pricepoint
        api_credits = data_status['credit_count'] # the number of credits spent on the last request
        
        # create a new pricepoint and save it to the SQLite db
        new_pricepoint = Price(datetime=api_timestamp, price=price, volume_24=volume_24, runid=current_run_id)
        id = new_pricepoint.save()

        # display what we have just saved
        print(f'request number: {s} - added {price} at {api_timestamp} - 24 hour volume: {volume_24} - credits used: {api_credits}')      

    except (ConnectionError, Timeout, TooManyRedirects) as e:
        print(e)
    time.sleep(query_interval)
print('finished')
run_id: 13 - timestamp: 2021-04-25 02:01:57.558206 - interval: 5 - number of queries: 200
request number: 0 - added 50119.3 at 2021-04-25T00:01:57.379Z - 24 hour volume: 48446424742.4 - credits used: 1
request number: 1 - added 50119.3 at 2021-04-25T00:02:02.561Z - 24 hour volume: 48446424742.4 - credits used: 1
request number: 2 - added 50119.3 at 2021-04-25T00:02:07.742Z - 24 hour volume: 48446424742.4 - credits used: 1
request number: 3 - added 50119.3 at 2021-04-25T00:02:12.917Z - 24 hour volume: 48446424742.4 - credits used: 1
request number: 4 - added 50115.2 at 2021-04-25T00:02:18.109Z - 24 hour volume: 48402342791.0 - credits used: 1
request number: 5 - added 50115.2 at 2021-04-25T00:02:23.692Z - 24 hour volume: 48402342791.0 - credits used: 1

Step #3 Querying the API

We can use the peewee command object.select to query the data from our SQLite tables. First, we will check for the number of runs.

query = Run.select().where(Run.id==current_run_id)
run_overview = pd.DataFrame(list(query.dicts()))
run_overview

Next, let’s select the price quotes for this id and plot them.

query = Price.select().where(Price.runid==current_run_id)
df_prices = pd.DataFrame(list(query.dicts()))
print(df_prices)

from pandas.plotting import register_matplotlib_converters
import matplotlib.dates as mdates
register_matplotlib_converters()
fig, ax = plt.subplots(figsize=(10, 8))
plt.title('BTC prices from the coinmarketcap API', fontsize=14)
datetimes = pd.to_datetime(df_prices['datetime'])

sns.lineplot(data=df_prices, x=datetimes, y="price")
#ax.set_xlim([df_prices['datetime'].min(),df_prices['datetime'].max()])
#ax.xaxis.set_major_locator(mdates.MinuteLocator())
plt.xticks(rotation=75)
price quotes obtained from the coinmarketcap api

Summary

Congratulations! You have it to the end of this Coinmarketcap API Python example. You have learned how to stream bitcoin price data in real-time using the Coinmarketcap API. We stored the data in an SQLite database using the Peewee ORM framework. Similarly, other APIs can be used to retrieve prices on a variety of assets, such as stocks or index prices.

If the tutorial helped you or you have any comments, feel free to let me know in the comments. And if you are interested in working with cryptocurrency price data, you might want to check out my recent article on color-coded bitcoin price charts.

Author

  • Hi, I am a Zurich-based Data Scientist with a passion for Machine Learning and Investing. After completing my Ph.D. in Business Informatics at the University of Bremen, I started working as a Machine Learning Consultant for the swiss consulting firm ipt. When I'm not working on use cases for our clients, I work on my own analytics projects and report on them in this blog.

Follow Florian Müller:

Data Scientist & Machine Learning Consultant

Hi, I am a Zurich-based Data Scientist with a passion for Machine Learning and Investing. After completing my Ph.D. in Business Informatics at the University of Bremen, I started working as a Machine Learning Consultant for the swiss consulting firm ipt. When I'm not working on use cases for our clients, I work on my own analytics projects and report on them in this blog.

Leave a Reply