retrieving data from coinmarketcap API using Python

Streaming Bitcoin Prices in Real-time via the Coinmarketcap API using Python

Streaming price data lays the foundation for many exciting machine learning applications in finance, such as generating trading signals for buy and sell orders. This tutorial shows how to stream bitcoin price quotes in real-time via a public API and save the data to a local SQLite database. Such use cases use real-time price data from a REST API endpoint. In this example, we will use Python to query the API from Coinmarketcap.com.

The rest of this article is structured as follows: Before getting your hands dirty in the coding part, you first need to obtain a free API key from Coinmarketcap. We will then use this key when we make calls to stream price quotes. We use Peewee, a lightweight SQLite framework to store the retrieved data to a local SQLite database. Finally, I show how you can query the obtained price quotes from our local database and use them later on, for example, to train a price prediction model.

Acquiring an API Key from Coinmarketcap

Coinmarketcap is one of the most popular websites in the crypto space. It provides price data, charts, and other information for a wide range of cryptocurrencies such as Bitcoin and Ethereum. Sending requests to the Coinmarketcap API requires an API key. To acquire your API key, you need to register an account. During the registration process, you will have the choice between different pricing plans. Fortunately, there is a free plan for personal use. It is a bit limited in terms of the number of API requests, but to get familiar with the API is more than sufficient.

coinmarketcap api pricing plans

After login into your account, you can display the API Key under the “Overview” section. For the sample code to work, you need to create a YAML file in the same folder where your Python project files are located, named coinmarketcap.yml. Insert your code into this file as follow:

key: "your coinmarketcap api key"

I’ll later show how you can import and access this key from your python code.

Each call to the Coinmarketcap API consumes credits. The number of calls and data we can retrieve via API is limited to 100 credits for the free plan. You can see how many of your daily and monthly credits are still available on the overview page.

coinmarketcap api key
Overview page of the coinmarketcap API

Python Example: Streaming Quotes from the Coinmarketcap API

In the following, we implement a regular call to the Coinmarketcap API. To do this, we create a loop that sends regular calls to the REST endpoint and returns price data in JSON format. After each call to the API, we create an object for the price point and store it in a local SQLite database using the Peewee framework.

Prerequisites

Before beginning the coding part, please make sure that you have set up your Python 3 environment and required packages. If you don’t have a Python environment available yet, you can follow the steps in this article to set up the Anaconda Python environment.

Also, make sure you install the required packages. In this tutorial, we will be working with the following standard packages: 

In addition, for visualization purposes, we will use Seaborn. To interact with the SQLite DB, we will use Peewee, a lightweight Object-Relational-Mapper (ORM) framework that lets us interact with SQLite in an object-oriented and more convenient way.

You can install packages using console commands:

  • pip install <package name>
  • conda install <package name> (if you are using the anaconda packet manager)

Step #1 Create a Relational Model in SQLite

We begin by defining a relational data model in SQlite, in which we will store the obtained price quotes. For this purpose, we will use Peewee, which is an object-oriented framework that allows us to create objects and persist them to the SQLite database. Our model contains two tables:

  • Run: A table in which we will store an ID and a timestamp, that makes it easier to identify past runs. Everything we start querying data from the API, we will add a new runid to this table.
  • The second table is called Price and contains the price quotes for a predetermined interval. We will store the price quotes in this table, and reference the runid via a foreign key.
from peewee import *
from datetime import date, timedelta, datetime
import numpy as np 
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import time
import json
import yaml
from requests import Request, Session
from requests.exceptions import ConnectionError, Timeout, TooManyRedirects

# persist information
db = SqliteDatabase('assets.db')

class BaseModel(Model):
    class Meta:
        database = db

class Run(BaseModel): # This model uses the "assets.db" database.
    timestamp = DateTimeField(unique=True)
    symbol = CharField()
    id = AutoField()

class Price(BaseModel): # This model uses the "assets.db" database.
    datetime = DateTimeField(unique=True)
    volume_24 = FloatField()
    price = FloatField()
    runid = ForeignKeyField(Run, backref='prices')
         
# By default, Peewee includes an IF NOT EXISTS clause when creating tables. 
def create_tables():
    with db:
        db.create_tables([Price])
        db.create_tables([Run])

def drop_tables():
    with db:
        db.drop_tables([Price])
        db.drop_tables([Run])
        
create_tables()

#drop_tables()    

If you want to drop the tables again, you can call our custom drop_tables function.

Step #2 Streaming Price Quotes

Once we have created the relational data model, we can define a request to the Coinmarketcap API and send a streaming request. Coinmarketcap offers different APIs. The one we will be using is cryptocurrency/listings/latest. This API provides the latest price data for several cryptocurrencies.

Before we query the API, we need to define basic parameters such as the conversion currency and the “limit.” We will select USD as convert and set the limit to 1. This value will lead to the API only returning the price of Bitcoin. A value of 2 will additionally also return the price of Ethereum (ETH). Higher values will give back additional cryptocurrencies depending on their market cap ranking.

I have set the query interval to 10 seconds, but you can change it to any other value. However, be aware that you will deplete your free credits quickly if you request data by the second. I have limited the number of price quotes retrieved to 100. If you want to retrieve price quotes for more extended periods, increase the timeframe.

Once we have defined these settings, we can query the API.

symbol='BTCUSD'
query_interval = 10 # in seconds
query_number = 100 # the number of queries after which the API stops

# load the API key from our local file
with open(r'C:/Users/Flo/relataly-public-python-tutorials/coinmarketcap.yml') as file:
    apikey = yaml.load(file, Loader=yaml.FullLoader)['key']

# Define some essential API parameters
# Coinmarketcap API for latest market ticker quotes and averages for cryptocurrencies and exchanges.
# https://coinmarketcap.com/api/documentation/v1/#operation/getV1CryptocurrencyListingsLatest
url = 'https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest'
parameters = {
  'start':'1',
  'limit':'1',
  'convert':'USD'
}
headers = {
  'Accepts': 'application/json',
  'X-CMC_PRO_API_KEY': apikey,
}

session = Session()
session.headers.update(headers)

# create a new run and save it to our local SQLite DB
run_timestamp = datetime.now()
run = Run(symbol=symbol, timestamp = run_timestamp)
run.save()
current_run_id = run.id 
print(f'run_id: {current_run_id} - timestamp: {run_timestamp} - interval: {query_interval} - number of queries: {query_number}')

# query the coinmarketcap API every x seconds
for s in range(0, query_number):
    try:
        response = session.get(url, params=parameters)
        data = json.loads(response.text)
        #print(data)
        
        # response - quote
        data_quote = data['data'][0]['quote']['USD']
        price = np.round(data_quote['price'], 1) # the price
        volume_24 = np.round(data_quote['volume_24h'], 1) # the volume in the last 24 hours
        
        # response - status
        data_status = data['status']
        api_timestamp = data_status['timestamp'] # the timestamp of the pricepoint
        api_credits = data_status['credit_count'] # the number of credits spent on the last request
        
        # create a new pricepoint and save it to the SQLite db
        new_pricepoint = Price(datetime=api_timestamp, price=price, volume_24=volume_24, runid=current_run_id)
        id = new_pricepoint.save()

        # display what we have just saved
        print(f'request number: {s} - added {price} at {api_timestamp} - 24 hour volume: {volume_24} - credits used: {api_credits}')      

    except (ConnectionError, Timeout, TooManyRedirects) as e:
        print(e)
    time.sleep(query_interval)
print('finished')
status response returned by the coinmarketcap API

Step #3 Querying the API

We can use the peewee “select” SQL command to query the data from our SQLite tables. First, we will check for the number of runs.

query = Run.select().where(Run.id==current_run_id)
run_overview = pd.DataFrame(list(query.dicts()))
run_overview

Next, let’s select the price quotes for this id and plot them.

query = Price.select().where(Price.runid==current_run_id)
df_prices = pd.DataFrame(list(query.dicts()))
print(df_prices)

from pandas.plotting import register_matplotlib_converters
import matplotlib.dates as mdates
register_matplotlib_converters()
fig, ax = plt.subplots(figsize=(10, 8))
plt.title('BTC prices from the coinmarketcap API', fontsize=14)
datetimes = pd.to_datetime(df_prices['datetime'])

sns.lineplot(data=df_prices, x=datetimes, y="price")
#ax.set_xlim([df_prices['datetime'].min(),df_prices['datetime'].max()])
#ax.xaxis.set_major_locator(mdates.MinuteLocator())
plt.xticks(rotation=75)
price quotes obtained from the coinmarketcap api

Summary

Congratulations! You have made it to the end of this Coinmarketcap API Python example. In this article, you have learned how to stream bitcoin price data in real-time using the Coinmarketcap API. We stored the data in an SQLite database using the Peewee ORM framework. After reading this article, you are well-prepared to use other APIs to retrieve prices on various assets, such as stocks or stock market indexes.

If the tutorial helped you or you have any comments, please let me know in the comments. And if you are interested in working with cryptocurrency price data, check out my recent article on color-coded bitcoin price charts.

Author

  • Hi, I am Florian, a Zurich-based consultant for AI and Data. Since the completion of my Ph.D. in 2017, I have been working on the design and implementation of ML use cases in the Swiss financial sector. I started this blog in 2020 with the goal in mind to share my experiences and create a place where you can find key concepts of machine learning and materials that will allow you to kick-start your own Python projects.

Leave a Reply