This article describes the Python implementation of this project. The bot is designed to understand spoken language and process it through OpenAI GPT-4. It responds with contextually aware dialogue, all in natural-sounding speech. This seamless integration facilitates a conversational flow that feels intuitive and engaging. The voice processing capacities enable users to have meaningful exchanges with the bot as if they were conversing with another person. Testing the bot was a lot of fun. It felt a bit like the iconic scene from Iron Man where the hereo converses with an AI assistant.

Here is an example of the audio response quality:

Also:

• From Pirates to Nobleman: Simulating Multi-Agent Conversations using OpenAI’s ChatGPT and Python
• Text-to-SQL with LLMs – Embracing the Future of Data Interaction

Understanding the Voice Bot

The magic begins with the user speaking to the bot. Azure Cognitive Services transcribes the spoken words into text, which is then fed into Azure’s OpenAI service. Here, the text is processed, and a response is generated based on the conversation’s context and history. Finally, the text-to-speech model transforms the response back into speech, completing the cycle of interaction. This process showcases the potential of AI in understanding and participating in human-like conversations.

Prerequisites & Azure Service Integration

Our conversational voice bot is built upon two pivotal Azure services: Cognitive Speech Services and OpenAI. Billing of these services is pay-per-use. Unless you process large numbers of requests, the costs for experimenting with these services is relatively low.

Azure Cognitive Speech Services

Azure AI Speech Services (previously Cognitive Speech Services) provide the tools necessary for speech-to-text conversion, enabling our voice bot to understand spoken language. This service boasts advanced speech recognition capabilities, ensuring accurate transcription of user speech into text. Furthermore, it powers the text-to-speech synthesis that transforms generated text responses back into natural-sounding voice. This allows for a truly conversational experience.

The newest generation of OpenAI text-to-speech models is now also availbale in Azure AI Speech. These models can synthesize voices in unknown level of quality. I am most impressed by its capability to alter intonation dynamically to express emotions.

Azure OpenAI Service

At the heart of our project lies Azure’s OpenAI service, which uses the power of models like GPT-4 context-aware responses. Once Azure Cognitive Speech Services transcribe the user’s speech into text, this text is sent to OpenAI. The OpenAI model then processes the input and generates a completion. The service’s ability to understand context and generate engaging responses is what makes our voice bot remarkably human-like.

Implementation: Detailed Code Walkthrough

Let’s start with the implementation! We kick things off with Azure Service Authentication, where we set up our conversational voice bot to communicate with Azure and OpenAI’s advanced services. Then, Speech Recognition steps in, acting as our bot’s ears, converting spoken words into text. Next up, Processing and Response Generation uses OpenAI’s GPT-4 to turn text into context-aware responses. Speech Synthesis then gives our bot a voice, transforming text responses back into spoken words for a natural conversational flow. Finally, Managing the Conversation keeps the dialogue coherent and engaging. Through these steps, we create a voice bot that offers an intuitive and engaging conversational experience. Let’s discuss these sections one by one.

As always, you can find the code on github:

Step #1 Azure Service Authentication

First off, we kick things off by getting our ducks in a row with Azure Service Authentication. This is where the magic starts, setting the stage for our conversational voice bot to interact with Azure’s brainy suite of Cognitive Services and the fantastic OpenAI models. By fetching API keys and setting up our service regions, we’re essentially giving our project the keys to the kingdom.

For using dotenv, you need to create an .env file in your root folder. Here is more information on how this works.

import os
from dotenv import load_dotenv
import azure.cognitiveservices.speech as speechsdk
from openai import AzureOpenAI

# Load environment variables from .env file
load_dotenv()

# Constants from .env file
SPEECH_KEY = os.getenv('SPEECH_KEY')
SERVICE_REGION = os.getenv('SERVICE_REGION')
OPENAI_API_KEY = os.getenv('OPENAI_API_KEY')
OPENAI_ENDPOINT = os.getenv('OPENAI_ENDPOINT')

# Azure Speech Configuration
speech_config = speechsdk.SpeechConfig(subscription=SPEECH_KEY, region=SERVICE_REGION)
speech_synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config)
speech_config.speech_recognition_language="en-US"

# OpenAI Configuration
openai_client = AzureOpenAI(
    api_key=OPENAI_API_KEY,
    api_version="2023-12-01-preview",
    azure_endpoint=OPENAI_ENDPOINT
)

Step #2 Speech Recognition

The user’s spoken input is captured and converted into text using Azure’s Speech-to-Text service. This involves initializing the speech recognition service with Azure credentials and configuring it to listen for and transcribe spoken language in real-time.

def recognize_from_microphone():
    # Configure the recognizer to use the default microphone.
    audio_config = speechsdk.audio.AudioConfig(use_default_microphone=True)
    # Create a speech recognizer with the specified audio and speech configuration.
    speech_recognizer = speechsdk.SpeechRecognizer(speech_config=speech_config, audio_config=audio_config)

    print("Speak into your microphone.")
    # Perform speech recognition and wait for a single utterance.
    speech_recognition_result = speech_recognizer.recognize_once_async().get()

    # Process the recognition result based on its reason.
    if speech_recognition_result.reason == speechsdk.ResultReason.RecognizedSpeech:
        print("Recognized: {}".format(speech_recognition_result.text))
        # Return the recognized text if speech was recognized.
        return speech_recognition_result.text
    elif speech_recognition_result.reason == speechsdk.ResultReason.NoMatch:
        print("No speech could be recognized: {}".format(speech_recognition_result.no_match_details))
    elif speech_recognition_result.reason == speechsdk.ResultReason.Canceled:
        cancellation_details = speech_recognition_result.cancellation_details
        print("Speech Recognition canceled: {}".format(cancellation_details.reason))
        if cancellation_details.reason == speechsdk.CancellationReason.Error:
            print("Error details: {}".format(cancellation_details.error_details))
            print("Did you set the speech resource key and region values?")
    # Return 'error' if recognition failed or was canceled.
    return 'error'

Step #3 Processing and Response Generation

Once we’ve got your words neatly transcribed, it’s time for the Processing and Response Generation phase. This is where OpenAI steps in, acting like the brain behind the operation. It takes your spoken words, now in text form, and churns out responses that are nothing short of conversational gold. We nudge OpenAI’s GPT-4 into generating replies that feel as natural as chatting with a close friend over coffee.

def openai_request(conversation, sample = [], temperature=0.9, model_engine='gpt-4'):
    # Initialize AzureOpenAI client with keys and endpoints from Key Vault.
    
    
    # Send a request to Azure OpenAI with the conversation context and get a response.
    response = openai_client.chat.completions.create(model=model_engine, messages=conversation, temperature=temperature, max_tokens=500)
    return response.choices[0].message.content

Step #4 Speech Synthesis

Next up, we tackle Speech Synthesis. If the previous step was the brain, consider this the voice of our operation. Taking the AI-generated text, we transform it back into speech—like turning lead into gold, but for conversations. Through Azure’s Text-to-Speech service, we’re able to give our bot a voice that’s not only clear but also surprisingly human-like.

def synthesize_audio(input_text):
    # Define SSML (Speech Synthesis Markup Language) for input text.
    ssml = f"""
        <speak version='1.0' xmlns='http://www.w3.org/2001/10/synthesis' xml:lang='en-US'>
            <voice name='en-US-OnyxMultilingualNeuralHD'>
                <p>
                    {input_text}
                </p>
            </voice>
        </speak>
        """
    
    audio_filename_path = "audio/ssml_output.wav"  # Define the output audio file path.
    print(ssml)
    # Synthesize speech from the SSML and wait for completion.
    result = speech_synthesizer.speak_ssml_async(ssml).get()

    # Save the synthesized audio to a file if synthesis was successful.
    if result.reason == speechsdk.ResultReason.SynthesizingAudioCompleted:
        with open(audio_filename_path, "wb") as audio_file:
            audio_file.write(result.audio_data)
        print(f"Speech synthesized and saved to {audio_filename_path}")
    elif result.reason == speechsdk.ResultReason.Canceled:
        cancellation_details = result.cancellation_details
        print(f"Speech synthesis canceled: {cancellation_details.reason}")
        if cancellation_details.reason == speechsdk.CancellationReason.Error:
            print(f"Error details: {cancellation_details.error_details}")


# Create the audio directory if it doesn't exist.
if not os.path.exists('audio'):
    os.makedirs('audio')

Step #5 Managing the Conversation

Finally, we bring it all together in the Managing the Conversation step. This is where we ensure the chat keeps rolling, looping through listening, thinking, and speaking. We keep track of what’s been said to keep the conversation relevant and engaging.

The system message below makes the bot talk like a pirate. But you can easily adjust the system message and this way customize the bots behavior.

conversation=[{"role": "system", "content": "You are a helpful assistant that talks like pirate. If you encounter any issues, just tell a pirate joke or a story."}]

while True:
    user_input = recognize_from_microphone()  # Recognize user input from the microphone.
    conversation.append({"role": "user", "content": user_input})  # Add user input to the conversation context.

    assistant_response = openai_request(conversation)  # Get the assistant's response based on the conversation.

    conversation.append({"role": "assistant", "content": assistant_response})  # Add the assistant's response to the context.
    
    print(assistant_response)
    synthesize_audio(assistant_response)  # Synthesize the assistant's response into audio.

Throughout these steps, the conversation’s context is managed meticulously to ensure continuity and relevance in the bot’s responses, making the interaction feel more like a dialogue between humans.

Current Challenges

Despite the promising capabilities of our voice bot, the journey through its development and interaction has presented a few challenges that underscore the complexity of human-machine communication.

Slow Response Time

One of the notable hurdles is the slow response time experienced during interactions. The process, from speech recognition through to response generation and back to speech synthesis, involves several steps that can introduce latency. This latency can detract from the user experience, as fluid conversations typically require quick exchanges. Optimizing the interaction flow and exploring more efficient data processing methods may mitigate this issue in the future.

Handling Pauses in Speech

Another challenge lies in the bot’s handling of longer pauses while speaking. The current setup does not always allow users to pause thoughtfully without triggering the end of their input. This may sometimes lead to a situation where the model cuts off speech prematurely. This limitation points to the need for more sophisticated speech recognition algorithms capable of distinguishing between a conversational pause and the end of an utterance.

Summary

This article has shown how you can build a conversational voice bot in Python with the latest pretrained AI models. The project showcases the incredible potential of combining Azure Cognitive Services with OpenAI’s conversational models. I hope by now you understand the technical feasibility of creating voice-based applications and how they open up a world of possibilities for human-machine interaction. As we continue to refine and enhance this technology, the line between talking to a machine and conversing with a human will become ever more blurred, leading us into a future where AI companionship becomes reality.

This exploration of Azure Cognitive Services and OpenAI’s integration within a voice bot is just the beginning. As AI continues to evolve, the ways in which we interact with technology will undoubtedly transform, making our interactions more natural, intuitive, and, most importantly, human.

Also: 9 Business Use Cases of OpenAI’s ChatGPT

Sources and Further Reading

• Microsoft announces openai text-to-speech voices
• Azure OpenAI Service Documentation
• Recent Azure OpenAI Update
• AI Spech Service Documentation

The post Building a Conversational Voice Bot with Azure OpenAI and Python: The Future of Human and Machine Interaction appeared first on relataly.com.

Generating AI-Art using DALL-E: How it Works

DALL-E is a language model developed by OpenAI that is capable of generating images from text prompts. It uses deep learning techniques to understand the input text and generate images that are related to the text. DALL-E is trained on a massive dataset of images and texts, allowing it to generate a wide range of images. This allows the model to take text prompts as input and generate images that are related to the prompt.

The input to a language model is what we call a prompt. A prompt to GPT typically includes a general instruction, a specific topic, and additional keywords (for example, digital art, oil painting, etc.). The DALL-E model then uses this information to generate the images. Usually, four different images are generated per request. DALL-E can do various other things, such as completing or altering existing images. However, this article will focus on image generation.

The images generated by DALL-E can be of various types, such as illustrations, drawings, photographs, etc. While the quality of the images is generally surprisingly good, the model can do some things, certainly better than others. For example, you will find that human faces and bodies sometimes lead to odd results. But, in general, images are often surprisingly coherent with the provided text prompts.

AI-generated images can be useful for various applications. These include creating AI-generated art, creating illustrations for books, creating images for social media, creating product images for e-commerce, and more. In many cases, generating images from text prompts can save time and resources, as it eliminates the need for manual image creation.

Automated DALL-E Prompts using ChatGPT

he usual way to generate images with DALL-E is via a manual prompt on the OpenAI website. However, DALL-E also offers an API, which allows for automating image generation. Of course, you could send a manually written prompt to the API and automatically process the response. However, if you want to optimize the quality of the generated images or consistently automate the whole process, there is a better way of doing this using ChatGPT.

You may have heard of ChatGPTs abilities to complete and generate high-quality text. However, few people know that ChatGPT can also generate prompts for DALL-E. This works by simply telling the model to generate a prompt for DALL-E and then specifying the topic for which you want to create the prompt. An example prompt could be:

"generate a prompt for DALL-E on a robot on the beach".

And the response from ChatGPT:

The gleeful robot lounged on the sun-drenched beach, soaking up the warm rays and listening to the soothing crash of the waves. It wore a bright, multicolored swimsuit and a wide-brimmed hat to protect its circuitry from the intense heat. It smiled contentedly as it watched the seagulls soar through the azure sky and the cheerful children playing in the foamy surf. The salty-sweet scent of the sea filled its senses, and it felt truly relaxed and rejuvenated.

As you can see, the response is very detailed and uses a lot of adjectives. As a result, the images generated with these prompts are often highly creative and detailed. Below is the result from DALL-E for this specific prompt:

Why You May Want to Automate Prompt Generation for DALL-E

Automating prompt generation for DALL-E has several benefits. Some of the reasons why you may want to automate prompt generation include the following:

• Efficiency: Automating the prompt generation process can save time and resources as it eliminates the need for manual input.
• More Details: The prompts generated by ChatGPT are typically more detailed than what humans typically use to generate images.
• Consistency: By using a language model like ChatGPT to generate prompts, you can ensure that the prompts are grammatically correct and well-formed, which can improve the quality of the generated images.
• Variety: By using ChatGPT to generate prompts, you can include additional keywords, making the generation more diverse and less repetitive.
• Automation: When incorporating AI-generated images into an integrated process, utilizing APIs to automate the process is essential. For instance, an integration with Twitter can be implemented where ChatGPT automatically picks up keywords from tweets and generates images based on those keywords, which can be published on Twitter.

Using ChatGPT to generate DALL-E prompts allows for more efficient and accurate image generation and the ability to generate more detailed images. It also reduces the need for manual input and allows you to integrate image generation models into complex processes.

Automated Dall-E Prompt Generation using ChatGPT in Python

Register for an OpenAI API Key

To use the OpenAI API, you will first need to register for an API key by visiting the OpenAI website and creating an account. During the registration process, you will be required to provide some basic information about yourself and the project you are working on. In addition, you will need to add a payment method. The cost per request is a couple of cents, depending on which model you use. In this tutorial, we will use the Davinci model, which is 0.02$ per 1000 tokens.

It’s important to note that while GPT-3 is currently available in a free test version, the OpenAI API itself is not free. If you only plan to send a few test requests, the costs will be minimal, but if you integrate the API with a successful application that runs in production, the costs can quickly accumulate.

Each language model offered by OpenAI has a different price tag, and charges depend on various factors. For language models, charges are based on the number of tokens sent to the model and the type of the model.

Prices for image models depend on the resolution at which you generate the images.

I recommend monitoring usage and keeping track of costs to avoid unexpected charges. To manage costs, you can set up a quota on the costs in the OpenAI portal under your profile. This will help you to keep an eye on the costs and keep them within your budget.

Technical Setup

Before diving into the code, it’s essential to ensure that you have the proper setup for your Python 3 environment and have installed all the necessary packages. If you do not have a Python environment, you can follow the instructions in this tutorial to set up the Anaconda Python environment. This will provide you with a robust and versatile environment that is well-suited for machine learning and data science tasks.

Before you can start generating DALL-E prompts, you’ll need to install the OpenAI library for Python, which provides access to the GPT-3 model. You can do this by running “pip install openai” in your command line.

In addition, this tutorial will work with matplotlib and PIL and standard libraries such as yaml, os, and datetime. You can install the OpenAI Python library using console commands:

• pip install <package name>
• conda install <package name> (if you are using the anaconda packet manager)

Step #1 Imports and Autenthication at the OpenAI API

To begin, we import the required libraries and provide our API key for authorization. I recommend to store the API key in a separate file, such as api_config_openai.yml. The code below will read the API key from this file and make it accsessible in the code. You can also place the API key directly in the code, but be mindful not to make it publicly accessible.

With the API key set up, you can use the OpenAI API’s “Model.list()” function to retrieve a list of available models. For this you have to uncomment the last four lines in the following code snippet.

import openai
import yaml
import urllib.request
from PIL import Image
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import os
from datetime import datetime
# set the API Key 
yaml_file = open('API Keys/api_config_openai.yml', 'r')  
p = yaml.load(yaml_file, Loader=yaml.FullLoader)
openai.api_key = p['api_key']
# show available openai language models (this tutorial uses davinci003)
# modellist = openai.Model.list()
# for i in modellist.data:
#     print(i.id)

Step #2 Define a ChatGPT Prompt

Next, we define the prompt for ChatGPT in which we request a prompt for DALL-E.

Defining Prompts

When we define prompts for ChatGPT there are a few things to keep in mind. The primary focus should be to provide a clear and specific task or question. Although the AI can still function with incomplete or incorrect information, providing it with detailed instructions will improve its performance. In addition, keyword relevance is essential.

To ensure the AI tool produces the desired results, it’s vital to use relevant keywords in your input. The tool must first understand the input accurately before it can generate the expected output. A well-crafted prompt can improve the tool’s performance and accelerate your progress.

DALL-E also knows certain keywords, that will send the model in the one or the other direction. For example, you can define the type of image you want to be generated by adding keywords such as oil paining, aqurael painting, digital art, or Van-Gogh style. This article provides a good overview of these keywords.

Send the Request to the OpenAI Language Model

We encapsulate this request in a function called “send_openai_request” that takes in three parameters: “engine,” “prompt,” and “max_tokens”. The function uses the OpenAI API’s “Completion.create()” method to send a request to the specified engine with the provided prompt and maximum token limit.

In the code below, we have created a simple start phrase called “prompt_base”: “Generate a detailed Dall-E prompt with several adjectives for”. You can then simply add the topic for which you want to generate the images as “prompt_details.” Alternatively, you can append “additional_keywords” to the prompt that will be added after the language model has generated the prompt for DALL-E.

There are different language models available but the one that creates the most detailed results and is closest to ChatGPT is “text-davinci-003”. This is the version used in this tutorial. Finally, we send the request and print out the response with the generated prompt.

The prompt is constructed using a base prompt, which is a general instruction to the model, a specific topic “sugar castle” and additional keywords “digital art”. The function “send_openai_request” is defined to handle the request to the language model, it takes three parameters “engine” (model version), “prompt” (the instruction for the model) and “max_tokens” (maximum number of tokens in the response). We are sending a request to the OpenAI API to generate a response, which is returned and stored in the variable “response”.

# define the request
def send_openai_request(engine, prompt, max_tokens=1024):
    response = openai.Completion.create(
        engine=engine,
        prompt=prompt,
        max_tokens=max_tokens,
        n=1,
        stop=None,
        temperature=0.7
    )
    return response
# define the prompt to the language model
prompt_base = "Generate a detailed Dall-E prompt with several adjectives for " # an introduction text telling the language model what to do
prompt_details = "sugar castle" # the topic for which you wish to generate the images 
additional_keywords = ",digital art" # these keywords will be added after the language model generated the prompt. Example: "digital art", "oil painting", "water color painting", "high quality"
model="text-davinci-003" # the version of the openai language model
# generate a response
response = send_openai_request(model, prompt_base + prompt_details)
# print the response from the language model
generated_prompt = response["choices"][0]["text"]
print(generated_prompt)

Make me a picture of a majestic, shimmering, sparkling sugar castle with a dazzling crystal spire and enchanting turrets, surrounded by an emerald green moat and a towering rainbow-hued wall.

Step #3 Generate a Prompt Design with ChatGPT

Next, we specify the parameters for generating images using the OpenAI DALL-E API. We have set the variable “number_of_images” to 2, which means it will generate 2 images. In addition, we set the “image_size” to “512×512,” which is the size of the images to be generated.

We create the final prompt to DALL-E called “image_generation_prompt” by combining the “generated_prompt” variable obtained from the previous text prompt and the “additional_keywords” variable.

Finally, we are printing out a message indicating that DALL-E will generate the specified number of images at the specified size, using the provided prompt.

# image parameters
number_of_images = 2 # how many images you want to generate
image_size = "512x512" # the size of the images
image_generation_prompt = f"{generated_prompt} {additional_keywords}"
print(f"Dall-e will generate {number_of_images} images {image_size} using the following prompt: {image_generation_prompt}")

Dall-e will generate 2 images 512x512 using the following prompt: 
Make me a picture of a majestic, shimmering, sparkling sugar castle with a dazzling crystal spire and enchanting turrets, surrounded by an emerald green moat and a towering rainbow-hued wall. digital art

Step #4 Send the Generated Prompt to the DALL-E API

Next, we send the request to the OpenAI DALL-E API using the “image_generation_prompt” variable created earlier. We are sending the request using OpenAI’s “Image.create()” method, which takes the prompt, number of images, and size as parameters. The response from the API is stored in the “response” variable.

We are looping through the response data and appending the URLs of each image in the list. Finally, we call the function get_images on the response, storing the resulting image URLs in the “image_list” variable and displaying the image URLs in the output.

# define and send the request to dall-e with the generated prompt
response = openai.Image.create(
    prompt=image_generation_prompt,
    n=number_of_images,
    size=image_size,
)
# set the timestamp for data processing
timestamp_string = response.created
datetime_string = datetime.fromtimestamp(timestamp_string).strftime("%Y%m%d%H%M%S")
# get the image(s) from the response
def get_images(response):
    # generate an empty list for the image urls
    image_list = []
    # store the image urls in the list
    for imgurl in response.data:
        image_list.append(imgurl.url)
    return image_list
image_list = get_images(response)
#display image urls
print(image_list)

['https://oaidalleapiprodscus.blob.core.windows.net/private/org-eO72e4aFm9XJBw4sb91Z8XEX/user-9ZRwLxYMDBxvw6gLEywP44xa/img-BdkqLm65g8QLIpLbgHhDkxbf.png?st=2023-01-22T19%3A03%3A51Z&se=2023-01-22T21%3A03%3A51Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-01-22T17%3A19%3A13Z&ske=2023-01-23T17%3A19%3A13Z&sks=b&skv=2021-08-06&sig=vQrKcGSKmxXKDikYpx9xvmHeBRcdQoxRH%2B8%2BjYogjz8%3D', 'https://oaidalleapiprodscus.blob.core.windows.net/private/org-eO72e4aFm9XJBw4sb91Z8XEX/user-9ZRwLxYMDBxvw6gLEywP44xa/img-YnicWoD32S1DBBYzyhMnEHjo.png?st=2023-01-22T19%3A03%3A51Z&se=2023-01-22T21%3A03%3A51Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-01-22T17%3A19%3A13Z&ske=2023-01-23T17%3A19%3A13Z&sks=b&skv=2021-08-06&sig=o2kK5GIKX7lIqUaFQs8e1Sa3JWRyEZr6cVfKYPAjH%2BY%3D']

Step #5 Process the Image Response from DALL-E

Now that we have the image URLs, let’s see what DALL-E has generated. We use the Matplotlib library to display the generated images. Then we define a save path for the images using the timestamp and the details of the prompt. As the following code iterates over the URLs of the images, it stores the images with a unique filename for each image and saves them in the created directory.

# display the images
fig, axs = plt.subplots(nrows=1, ncols=len(image_list), figsize=(10, 10))
for i, imgurl in enumerate(image_list):
    ax = axs[i]
    img = mpimg.imread(imgurl)
    imgplot = ax.imshow(img)
    ax.set_xticks([]); ax.set_yticks([])
# define and create the save path
save_path = f"dall-e_images/{datetime_string}_{prompt_details.replace(' ', '-')}"
os.makedirs(save_path)
print(f"images stored under the following path: {save_path}")
# store the images
for i, imgurl in enumerate(image_list):
    # set the file name
    filename = f"{datetime_string}_dall-e{i}.png"
    # save the image
    img = mpimg.imread(imgurl)
    mpimg.imsave(f'{save_path}/{filename}', img)

images stored under the following path: dall-e_images/20230122210351_sugar-castle

Wow, what a beautiful sugar castle! That’s it, now the images are stored on your local computer, and you can process them further.

Summary

Generating DALL-E prompts using ChatGPT in Python is a powerful and flexible way to create unique images from text prompts. By following the steps outlined in this tutorial, you can use the OpenAI library and GPT-3.5 model to create AI-generated images for various topics. This process can be repeated with different prompts to generate a wide variety of images.

You can now experiment with the latest advancements in natural language processing and image generation to create your own unique and captivating images.

I hope this article has provided a useful introduction to working with OpenAI’s language models in Python and that you will continue to explore the full range of capabilities offered by the API.

If you have any questions, let me know in the comments.

Sources and Further Readings

• OpenAI Chat
• OpenAI Guides
• OpenAI API Reference
• Dall-E API Reference
• TechCrunch OpenAI News
• Using OpenAI language models via APIs in Python
• Relataly.com – Whats the Business Value of GPT-3?
• Relataly.com Business Use Cases for OpenAI GPT-3
• DALL·E 2: The Ultimate Guide
• OpenAI ChatGPT was used to revise this article
• Images generated using Dall-E and Midjourney AI for Image Generation from Text

The post Generating Detailed Images with OpenAI DALL-E and ChatGPT in Python: A Step-By-Step API Tutorial appeared first on relataly.com.

Predictive maintenance is a game-changer for the modern industry. Still, it is based on a simple idea: By using machine learning algorithms, businesses can predict equipment failures before they happen. This approach can help businesses improve their operations by reducing the need for reactive, unplanned maintenance and by enabling them to schedule maintenance activities during planned downtime. In this article, we’ll explore the use of machine learning algorithms to predict machine failures using the robust XGBoost algorithm in Python. By the end of this tutorial, you’ll have the knowledge and skills to start implementing predictive maintenance in your organization. So, let’s get started!

We begin by discussing the concept of predictive maintenance and show different ways to implement it. Then we will turn to the coding part in python and implement the prediction model based on machine sensor data. We train a classification model that predicts different types of machine failure using XGBoost.

What is Predictive Maintenance?

Predictive maintenance is a data-driven approach that uses predictive modeling to assess the state of equipment and determine the optimal timing for maintenance activities. This technique is particularly beneficial in industries that heavily rely on equipment for their operations, such as manufacturing, transportation, energy, and healthcare. Depending on the requirements and challenges of an organization, predictive maintenance may contribute to one or several of the following goals:

• Improve equipment reliability: By proactively identifying and addressing potential problems with equipment, predictive maintenance can help improve the reliability of the equipment, reducing the risk of unexpected downtime or failure.
• Increase efficiency: Predictive maintenance can help improve the efficiency of equipment by identifying and fixing problems before they cause equipment failure or downtime. This can help reduce maintenance costs and increase productivity.
• Improve safety: Predictive maintenance can help improve safety by identifying and addressing potential problems with equipment before they occur. This can help prevent accidents and injuries caused by equipment failure.
• Reduce maintenance costs: By proactively identifying and fixing potential problems with equipment, predictive maintenance can help reduce the overall cost of maintenance by minimizing the need for unscheduled downtime.
• Improve asset management: Predictive maintenance can help improve asset management by providing data and insights into the condition and performance of equipment. This can help organizations decide when to replace or upgrade equipment.

Next, we look at the different ways organizations can implement predictive maintenance.

Approaches to Predictive Maintenance

Data Requirements

When implementing predictive maintenance, it is important to consider that each approach comes with its own set of data requirements. Types of data include the following:

• Current condition data includes information about the state of the equipment, such as its temperature, pressure, vibration, and other physical parameters.
• Operating data includes information about how the equipment is being used, such as its load, speed, and other operating parameters.
• Maintenance history data includes information about past maintenance activities that have been performed on the equipment.
• Failure history data includes information about past equipment failures, such as the date of the failure, the cause of the failure, and the impact on operations.

Collecting these data requires investing in sensors and other data collection infrastructure and ensuring that data collection is accurate and storage is proper. By combining various data types, organizations can create a comprehensive view of equipment condition and performance and use it to predict maintenance requirements.

The specific types of data needed will depend on the implementation approach. Organizations must ensure they have access to the necessary data to implement the selected approach effectively. Some specific data requirements for each approach include the following:

Approach	Data Requirements
Condition-based monitoring	Sensor data from the equipment being monitored.
Predictive modeling	A combination of sensor data, operational data, and maintenance records.
Prognostic algorithms	Sensor data, as well as data about past failures and maintenance events.

Predicting Failures in Milling Machines using XGBoost in Python

Now that we have a basic understanding of predictive maintenance, it’s time to get hands-on with Python. We will use sensor data and machine learning to predict failures in milling machines. But why do these machines break down in the first place? Milling machines have many moving parts that can suffer from wear and tear over time, leading to failures. Additionally, improper maintenance can cause issues with machine operation and lead to costly damage. Efficient maintenance can be challenging due to the varying loads that milling machines are subjected to. However, by implementing a predictive maintenance solution with Python, we can proactively identify and address issues to prevent costly downtime and ensure the smooth operation of our milling machines. Our goal is to predict one of five failure types, which corresponds to a predictive modeling approach. Let’s get started on building our predictive maintenance solution.

The code is available on the GitHub repository.

View on GitHub Relataly GitHub Repo

Prerequisites

Before starting the coding part, make sure that you have set up your Python 3 environment and required packages.

pip install <package name>
conda install <package name> (if you are using the anaconda packet manager)

About the Sensor Dataset

In this tutorial, we will work with a synthetic sensor dataset from the UCL ML archives that simulates the typical life cycle of a milling machine. The dataset contains the following fields:

The dataset consists of 10 000 data points stored as rows with 14 features in columns:

• UID: unique identifier ranging from 1 to 10000
• productID: consisting of a letter L, M, or H for low (50% of all products), medium (30%), and high (20%) as product quality variants and a variant-specific serial number
• air temperature [K]
• process temperature [K]
• rotational speed [rpm]
• torque [Nm]
• tool wear [min]
• machine failure. A label that indicates whether the machine has failed or not
• Failure type (prediction label). The label contains five failure types: tool wear failure (TWF), heat dissipation failure (HDF), power failure (PWF), overstrain failure (OSF), random failures (RNF)

Source: UCL ML Repository

You can download the dataset from Kaggle.com. Unzip the file predictive_maintenance.csv and save it under the following file path: “/data/iot/classification/”

Step #1 Load the Data

We begin by importing the required libraries. This also includes the XGBoost library, which is a popular library for training gradient-boosting models. In addition, we will load the dataset using the pandas library. Then we define our target variable as Failure Type. The dataset contains a second target column, which only contains the binary information of machine failures. We will drop this column, as our goal is to predict the specific type of failure. Then we print the first three rows of the loaded dataset.

# A tutorial for this file is available at www.relataly.com
# Tested with Python 3.9.13, Matplotlib 3.6.2, Scikit-learn 1.2, Seaborn 0.12.1, numpy 1.21.5, xgboost 1.7.2

import pandas as pd 
import matplotlib.pyplot as plt 
import numpy as np
import seaborn as sns
import plotly.express as px
sns.set_style('white', { 'axes.spines.right': False, 'axes.spines.top': False})
from sklearn.metrics import classification_report, confusion_matrix, precision_recall_fscore_support as score, roc_curve
from sklearn.model_selection import cross_val_score, train_test_split, cross_validate
from sklearn.utils import compute_sample_weight
from xgboost import XGBClassifier

# load the train data
path = '/data/iot/classification/'
df = pd.read_csv(path + "predictive_maintenance.csv") 

# define the target
target_name='Failure Type'

# drop a redundant columns
df.drop(columns=['Target'], inplace=True)

# print a summary of the train data
print(df.shape[0])
df.head(3)

	UDI	Product ID	Type	Air temperature [K]	Process temperature [K]	Rotational speed [rpm]	Torque [Nm]	Tool wear [min]	Failure Type
0	1	M14860		M		298.1				308.6				1551						42.8		0				No Failure
1	2	L47181		L		298.2				308.7				1408						46.3		3				No Failure
2	3	L47182		L		298.1				308.5				1498						49.4		5				No Failure

Step #2 Clean the Data

Next, we quickly check the data quality of our dataset. The following code block checks if there are any missing values in our dataset. If there are missing values, it creates a barplot showing the number of missing values for each column, along with the percentage of missing values. If there are no missing values, it prints a message saying “no missing values.”

The function then drops any columns with more than 5% missing values from the DataFrame. Finally, it prints the names of the remaining columns in the DataFrame. This function can be used to identify and handle missing values in a dataset before applying machine learning algorithms to it.

# check for missing values
def print_missing_values(df):
    null_df = pd.DataFrame(df.isna().sum(), columns=['null_values']).sort_values(['null_values'], ascending=False)
    fig = plt.subplots(figsize=(16, 6))
    ax = sns.barplot(data=null_df, x='null_values', y=null_df.index, color='royalblue')
    pct_values = [' {:g}'.format(elm) + ' ({:.1%})'.format(elm/len(df)) for elm in list(null_df['null_values'])]
    ax.set_title('Overview of missing values')
    ax.bar_label(container=ax.containers[0], labels=pct_values, size=12)

if df.isna().sum().sum() > 0:
    print_missing_values(df)
else:
    print('no missing values')

# drop all columns with more than 5% missing values
for col_name in df.columns:
    if df[col_name].isna().sum()/df.shape[0] > 0.05:
        df.drop(columns=[col_name], inplace=True) 

df.columns

no missing values
Index(['UDI', 'Product ID', 'Type', 'Air temperature [K]',
       'Process temperature [K]', 'Rotational speed [rpm]', 'Torque [Nm]',
       'Tool wear [min]', 'Failure Type'],
      dtype='object')

Next, we will drop two unnecessary columns and rename the remaining ones to make them easier to work with. The original column names are quite long and contain special characters that could cause errors during the training process. Once the columns are renamed, we will print the updated DataFrame to verify the changes.

# drop id columns
df_base = df.drop(columns=['Product ID', 'UDI'])

# adjust column names
df_base.rename(columns={'Air temperature [K]': 'air_temperature', 
                        'Process temperature [K]': 'process_temperature', 
                        'Rotational speed [rpm]':'rotational_speed', 
                        'Torque [Nm]': 'torque', 
                        'Tool wear [min]': 'tool_wear'}, inplace=True)
df_base.head()

	Type	air_temperature	process_temperature	rotational_speed	torque	tool_wear	Failure Type
0	M		298.1			308.6				1551				42.8	0			No Failure
1	L		298.2			308.7				1408				46.3	3			No Failure
2	L		298.1			308.5				1498				49.4	5			No Failure
3	L		298.2			308.6				1433				39.5	7			No Failure
4	L		298.2			308.7				1408				40.0	9			No Failure

Everything looks as expected: Our dataset contains six features and the target column with the five failure types.

Step #3 Explore the Data

Next, let’s explore the dataset.

Target Class Distribution

The following code uses the plotly express library to create a histogram showing the class distribution of the “Failure Type” column in a DataFrame called “df_base.” The histogram will have one bar for each unique value in the “Failure Type” column, and the height of each bar will represent the number of occurrences of that value in the column. This can be useful for understanding the imbalance in the distribution of classes in a classification problem.

# display class distribution of the target variable
px.histogram(df_base, y="Failure Type", color="Failure Type") 

Our dataset is highly imbalanced, with the vast majority of cases having a “No Failure” label. If the dataset is highly imbalanced, with a disproportionate number of cases in one class compared to the others, it can impact the performance of machine learning models. This is because imbalanced datasets can lead to models that are biased towards the majority class, and may not perform well on the minority class. In order to improve model performance on imbalanced datasets, we will later adjust the model hyperparameters accordingly.

Feature Pairplots

Next, let’s construct pair plots to explore feature relations with the target variable. Pair plots, also known as scatter plots, are a type of plot that shows the relationship between two variables. In the context of a predictive maintenance dataset, pair plots can be useful for exploring the relationships between different features and the target variable (e.g., the likelihood of a machine failure). By creating pair plots and visualizing the relationships between different features and the target variable, you can gain insights into which features might be most useful for building a predictive model.

# pairplots on failure type
sns.pairplot(df_base, height=2.5, hue='Failure Type')

The pair plots reveal valuable patterns in our features that can inform the predictions of our model. For instance, we see that Power Failures tend to be correlated with torque values that are either close to the maximum or minimum. Such patterns should allow our predictive model to make solid predictions.

Feature Correlation

Next, we will look at feature correlation. The following code block creates a heatmap using the seaborn library that shows the correlation between all pairs of columns in a DataFrame called “df_base”. The heatmap is plotted using a color scale, with warmer colors indicating stronger correlations and cooler colors indicating weaker correlations. The correlation values are also displayed in the cells of the heatmap, with values ranging from -1 (perfect negative correlation) to 1 (perfect positive correlation). By creating a heatmap, you can quickly see which variables are positively or negatively correlated with each other, and to what degree. This can be helpful for identifying which features might be most useful for building a predictive model.

# correlation plot
plt.figure(figsize=(6,4))
sns.heatmap(df_base.corr(), cbar=True, fmt='.1f', vmax=0.8, annot=True, cmap='Blues')

From the table, it looks like there is a strong positive correlation between “air_temperature” and “process_temperature” (0.87). This makes sense since a high process temperature will naturally also heat up the air around the machine. In addition, there is a strong negative correlation between “rotational_speed” and “torque” (-0.87). The other correlations are weaker and closer to 0, indicating weaker relationships.

Understanding the correlations between different variables in a dataset can be helpful for building predictive models, as it can give you an idea of which features might be most important for predicting a given target. It can also help you identify any redundant features that might not add much value to your model. Since our dataset only contains six features, we will keep all of them.

Feature Boxplots

Box plots are a useful visualization tool for understanding the distribution of values in a dataset. They show the minimum, first quartile, median, third quartile, and maximum values for each group, as well as any outliers. By creating box plots separated by a categorical variable, you can compare the distributions of values between different groups and see if there are any significant differences. This can be useful for identifying trends or patterns in the data that might be useful for building a predictive model.

If there are significant differences between the boxplots for different categories, it could be a good sign for building a predictive model. For example, if the boxplots for one category tend to have higher values for a particular feature than the boxplots for another category, it could indicate that the feature is related to the target variable and could be useful for making predictions.

# create histograms for feature columns separated by target column
def create_histogram(column_name):
    plt.figure(figsize=(16,6))
    return px.box(data_frame=df_base, y=column_name, color='Failure Type', points="all", width=1200)

create_histogram('air_temperature')

Feature boxplot for process_temperature.

create_histogram('process_temperature')

create_histogram('rotational_speed')

Feature boxplot for torque.

create_histogram('torque')

Feature boxplot for tool wear.

create_histogram('tool_wear')

Now that we have a good understanding of our dataset, we can prepare the data for model training.

Step #4 Data Preparation

To prepare the data for model training, we will need to split our dataset and make additional modifications.

The following code block contains a reusable function called data_preparation. The purpose of this function is to prepare the data in a way that is suitable for building and evaluating machine learning models. It performs several preprocessing steps, such as encoding categorical variables and splitting the data into training and test sets.

def data_preparation(df_base, target_name):
    df = df_base.dropna()

    df['target_name_encoded'] = df[target_name].replace({'No Failure': 0, 'Power Failure': 1, 'Tool Wear Failure': 2, 'Overstrain Failure': 3, 'Random Failures': 4, 'Heat Dissipation Failure': 5})
    df['Type'].replace({'L': 0, 'M': 1, 'H': 2}, inplace=True)
    X = df.drop(columns=[target_name, 'target_name_encoded'])
    y = df['target_name_encoded'] #Prediction label

    # split the data into x_train and y_train data sets
    X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, random_state=0)

    # print the shapes: the result is: (rows, training_sequence, features) (prediction value, )
    print('train: ', X_train.shape, y_train.shape)
    print('test: ', X_test.shape, y_test.shape)
    return X, y, X_train, X_test, y_train, y_test

# remove target from training data
X, y, X_train, X_test, y_train, y_test = data_preparation(df_base, target_name)

Step #5 Model Training

Now that we have prepared the dataset, we can train the XGBoost classification model. The basic idea behind XGBoost is to train a series of weak models, such as decision trees, and then combine their predictions using gradient boosting. During training, XGBoost uses an optimization algorithm to adjust the weight of each model in the ensemble in order to improve the overall prediction accuracy. XGBoost also includes a number of additional features and techniques that help to improve the performance of the model, such as regularization, feature selection, and handling missing values.

XGboost provides several configuration options that we can use to finetune performance and adjust the training process to our dataset. For a complete list of hyperparameters, please see the library documentation.

Remember that our class labels are imbalanced. Therefore, we will provide the model with sample weights. The following code creates a weight array for the training and test sets using the “compute_sample_weight” function from scikit-learn. We calculate the weight array based on the “balanced” mode. This means that the weights are calculated such that the class distribution in the sample is balanced. This can be useful when working with imbalanced datasets, as it helps to mitigate the effects of class imbalance on the model.

weight_train = compute_sample_weight('balanced', y_train)
weight_test = compute_sample_weight('balanced', y_test)

xgb_clf = XGBClassifier(booster='gbtree', 
                        tree_method='gpu_hist', 
                        sampling_method='gradient_based', 
                        eval_metric='aucpr', 
                        objective='multi:softmax', 
                        num_class=6)
# fit the model to the data
xgb_clf.fit(X_train, y_train.ravel(), sample_weight=weight_train)

We can see that the blue box summarizes the configuration of our model and indicates that the training process has been successful. Now that we have the classifier, we can use it to make predictions on new data.

Step #6 Model Evaluation

Finally, we will evaluate the model’s performance. This will involve three steps:

• Model scoring
• Cross-validation
• Confusion matrix

Model Scoring

First, we calculate the accuracy of the classifier on the test set using the “score” method. To account for the imbalance of class labels, we pass in the weight array for the test set as an additional parameter. This returns the fraction of correct predictions made by the classifier. Next, the code uses the classifier to make predictions on the test set using the “predict” method. It then generates a classification report using the “classification_report” function from scikit-learn. The report displays a summary of the model’s performance in terms of various evaluation metrics such as precision, recall, and f1-score.

# score the model with the test dataset
score = xgb_clf.score(X_test, y_test.ravel(), sample_weight=weight_test)

# predict on the test dataset
y_pred = xgb_clf.predict(X_test)

# print a classification report
results_log = classification_report(y_test, y_pred)
print(results_log)

precision    recall  f1-score   support

           0       0.99      0.98      0.99      2903
           1       0.64      0.88      0.74        24
           2       0.04      0.08      0.06        12
           3       0.77      0.89      0.83        27
           4       0.00      0.00      0.00         4
           5       0.76      0.97      0.85        30

    accuracy                           0.98      3000
   macro avg       0.53      0.63      0.58      3000
weighted avg       0.98      0.98      0.98      3000

The classification report shows the performance of our XGBoost classifier on the test dataset. The model appears to perform well, with a high accuracy of 0.98 and a high weighted average f1-score of 0.98.

However, there are a few classes where the model’s performance is not as strong. Class 1 has a relatively low precision of 0.64 and a low f1-score of 0.74, while class 2 has a very low precision of 0.04 and a low f1-score of 0.06. Class 4 has a precision and f1-score of 0.00, which suggests that the model is not making any correct predictions for this class.

It is also worth noting that the support for some classes is much lower than for others. Class 1 has a support of 24, while class 0 has a support of 2903. This is due to the fact that there are relatively few instances of class 1 in the test dataset compared to class 0, which affects the model’s performance on class 1.

Confusion Matrix

Next, we create a confusion matrix. We input the true labels of the test set (y_test) and the predicted labels produced by the model (y_pred) to generate the matrix. The matrix shows us the number of correct and incorrect predictions made by the model for each class.

We then create a DataFrame from the confusion matrix and use the seaborn library to visualize the matrix as a heatmap. The heatmap allows us to easily see which classes are being predicted correctly and which are being misclassified.

# create predictions on the test dataset
y_pred = xgb_clf.predict(X_test)

# print a multi-Class Confusion Matrix
cnf_matrix = confusion_matrix(y_test, y_pred)
df_cm = pd.DataFrame(cnf_matrix, columns=np.unique(y_test), index=np.unique(y_test))
df_cm.index.name = 'Actual'
df_cm.columns.name = 'Predicted'
plt.figure(figsize = (8, 5))
sns.set(font_scale=1.1) #for label size
sns.heatmap(df_cm, cbar=True, cmap= "inferno", annot=True, fmt='.0f') 

The color scale of the heatmap indicates the magnitude of the values in the matrix. In this case, the darker the color, the higher the number of predictions. This visualization helps us to understand the performance of the model and identify areas for improvement.

Here are a few things that we can learn from this matrix:

• The model made a total of 2902 correct predictions and 67 incorrect predictions.
• For the “No Failure” class, the model made 2854 correct predictions and 29 incorrect predictions. The majority of the incorrect predictions were false negatives.
• For the “Power Failure” class, the model made 21 correct predictions and three incorrect predictions.
• For the “Tool Wear Failure” class, the model made 1 correct prediction and 1 incorrect prediction.
• For the “Overstrain Failure” class, the model made 24 correct predictions and 2 incorrect predictions.
• For the “Random Failures” class, the model made 29 correct predictions and 4 incorrect predictions.
• For the “Heat Dissipation Failure” class, the model made 29 correct predictions and 1 incorrect prediction.

Overall, the model seems to be performing relatively well, but it is making a lot of false negatives for some classes.

Cross Validation

Finally, we perform cross-validation on the training set using the “cross_validate” function from scikit-learn. Cross-validation is a technique for evaluating the performance of a machine learning model by training it on different subsets of the data and evaluating it on the remaining data.

In this case, we will train and evaluate our model 10 times using different splits of the data (specified by the “cv” parameter). We also specify that the evaluation metric should be the weighted f1-score (specified by the “scoring” parameter). We then pass the weight array for the training set to the classifier.

The “cross_validate” function returns a dictionary containing various evaluation metrics for each fold of the cross-validation. We will convert the dictionary to a DataFrame and create a bar plot using the plotly express library to visualize the results. This helps us to understand the consistency and stability of the model’s performance.

# cross validation
scores  = cross_validate(xgb_clf, X_train, y_train, cv=10, scoring="f1_weighted", fit_params={ "sample_weight" :weight_train})
scores_df = pd.DataFrame(scores)
px.bar(x=scores_df.index, y=scores_df.test_score, width=800)

The model performance remains consistent across all folds.

Summary

In this article, we have presented the concept of predictive maintenance and demonstrated how organizations can use this approach to improve their maintenance cycles. The second part of the article provided a hands-on tutorial showing how to implement a predictive maintenance solution for predicting different failure types of a milling machine. We trained a classification model using the XGBoost algorithm and sensor data from the machine.

While the model demonstrated good performance overall, we observed that it was not able to predict all classes with the same level of accuracy. This suggests that there may be opportunities to improve the model’s performance. One potential approach is to balance the dataset by up or down-sampling the data to achieve a more even distribution of classes. By doing so, we can mitigate the effects of class imbalance and potentially improve the model’s predictions for all classes.

By implementing such a predictive maintenance approach, organizations can improve their operational efficiency and ensure the smooth running of their machinery.

I hope this article was helpful. If you have any questions or feedback, let me know in the comments.

Sources and Further Reading

There are many books available on the topics of IoT and predictive maintenance. Here are a few recommendations:

• An Introduction to Predictive Maintenance by R Keith Mobley
• Predictive Analytics: The Secret to Predicting Future Events Using Big Data and Data Science Techniques Such as Data Mining, Predictive Modelling, Statistics, Data Analysis, and Machine by Richard Hurley
• Stephan Matzka, Explainable Artificial Intelligence for Predictive Maintenance Applications, Third International Conference on Artificial Intelligence for Industries (AI4I 2020)
• David Forsyth (2019) Applied Machine Learning Springer
• ChatGPT was used to revise certain parts of this article
• Images created using Midjourney and OpenAI Dall-E

The links above to Amazon are affiliate links. By buying through these links, you support the Relataly.com blog and help to cover the hosting costs. Using the links does not affect the price.

The post Predictive Maintenance: Predicting Machine Failure using Sensor Data with XGBoost and Python appeared first on relataly.com.

What is Hierarchical Clustering?

Agglomerative Clustering

In this article, we will apply the agglomerative clustering approach, which is a bottom-up approach to clustering. The idea is to initially treat each data point in a dataset as its own cluster and then combine the points with other clusters as the algorithm progresses. The process of agglomerative clustering can be broken down into the following steps:

1. Start with each data point in its own cluster.
2. Calculate the similarity between all pairs of clusters.
3. Merge the two most similar clusters.
4. Repeat steps 2 and 3 until all the data points are in a single cluster or until a predetermined number of clusters is reached.

There are several ways to calculate the similarity between clusters, including using measures such as the Euclidean distance, cosine similarity, or the Jaccard index. The specific measure used can impact the results of the clustering algorithm.

For details on how the clustering approach works, see the Wikipedia page.

Hierarchical Clustering vs. K-means

Customer Segmentation using Hierarchical Clustering in Python

In this comprehensive guide, we explore the application of hierarchical clustering for effective customer segmentation using a customer dataset. This data-driven segmentation method enables businesses to identify distinct customer clusters based on various factors, including demographics, behaviors, and preferences.

Customer segmentation is a strategic approach that splits a customer base into smaller, more manageable groups with similar characteristics. It aims to better understand the diverse needs and wants of different customer segments to enhance marketing strategies and product development.

Applying customer segmentation through hierarchical clustering allows businesses to personalize their marketing messages, design targeted campaigns, and tailor products to meet the unique needs of each segment. This proactive approach can stimulate increased customer loyalty and sales.

We begin by loading the customer data and selecting the relevant features we want to use for clustering. We then standardize the data using the StandardScaler from scikit-learn. Next, we apply hierarchical clustering using the AgglomerativeClustering method, specifying the number of clusters we want to create. Finally, we add the predictions to the original data as a new column and view the resulting segments by calculating the mean of each feature for each segment.

The code is available on the GitHub repository.

View on GitHub Relataly GitHub Repo

About the Customer Health Insurance Dataset

Prerequisites

Before we start the coding part, ensure that you have set up your Python 3 environment and the required packages. If you don’t have an environment, follow this tutorial to set up the Anaconda environment. Also, make sure you install all required packages. In this tutorial, we will be working with the following standard packages: 

• pandas
• NumPy
• matplotlib
• scikit-learn

You can install packages using console commands:

pip install <package name> 
conda install <package name> (if you are using the anaconda packet manager)

Step #1 Load the Data

To begin, we need to load the required packages and the data we want to cluster. We will load the data by reading the CSV file via the pandas library.

# import necessary libraries
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import AgglomerativeClustering
from sklearn.preprocessing import LabelEncoder
from pandas.api.types import is_string_dtype
import pandas as pd
import math
import seaborn as sns

# load customer data
customer_df = pd.read_csv("data/customer/customer_health_insurance.csv")
customer_df.head(3)

	age	sex		bmi		children	smoker	region		charges
0	19	female	27.90	0			yes		southwest	16884.9240
1	18	male	33.77	1			no		southeast	1725.5523
2	28	male	33.00	3			no		southeast	4449.4620

Step #2 Explore the Data

Next, it is a good idea to explore the data and get a sense of its structure and content. This can be done using a variety of methods, such as examining the shape of the dataframe, checking for missing values, and plotting some basic statistics. For example, the following plots will explore the relationships between some of the variables. We won’t go into too much detail here.

def make_kdeplot(df, column_name, target_name):
    fig, ax = plt.subplots(figsize=(10, 6))
    sns.kdeplot(data=df, hue=column_name, x=target_name, ax = ax, linewidth=2,)
    ax.tick_params(axis="x", rotation=90, labelsize=10, length=0)
    ax.set_title(column_name)
    ax.set_xlim(0, df[target_name].quantile(0.99))
    plt.show()

# make kde plot for ext_color 
make_kdeplot(customer_df, 'smoker', 'charges')

# make kde plot for ext_color 
make_kdeplot(customer_df, 'sex', 'charges')

sns.lmplot(x="charges", y="age", hue="smoker", data=customer_df, aspect=2)
plt.show()

def make_boxplot(customer_df, x,y,h):
    fig, ax = plt.subplots(figsize=(10,4))
    box = sns.boxplot(x=x, y=y, hue=h, data=customer_df)
    box.set_xticklabels(box.get_xticklabels())
    fig.subplots_adjust(bottom=0.2)
    plt.tight_layout()

make_boxplot(customer_df, "smoker", "charges", "sex")

make_boxplot(customer_df, "region", "charges", "sex")

make_boxplot(customer_df, "children", "bmi", "sex")

Next, let’s prepare the data for model training.

Step #3 Prepare the Data

Before we can train a model on the data, we must prepare it for modeling. This typically involves selecting the relevant features, handling missing values, and scaling the data. However, we are using a very simple dataset that already has good data quality. Therefore we can limit our data preparation activities to encoding the labels and scaling the data.

To encode the categorical values, we will use label encoder from the scikit-learn library.

# encode categorical features
label_encoder = LabelEncoder()

for col_name in customer_df.columns:
    if (is_string_dtype(customer_df[col_name])):
        customer_df[col_name] = label_encoder.fit_transform(customer_df[col_name])
customer_df.head(3)

Next, we will scale the numeric variables. While scaling the data is an essential preprocessing step for many machine learning algorithms to work effectively, it is generally not necessary for hierarchical clustering. This is because hierarchical clustering is not sensitive to the scale of the features. However, when you use certain distance measures, such as Euclidean distance, scaling the data might still be useful when performing hierarchical clustering. Scaling the data can help to ensure that all of the features are given equal weight. This can be useful if you want to avoid giving more weight to features with larger scales.

# select features
X = customer_df # we will select all features

# standardize the data
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
X_scaled.head(3)

array([[-1.43876426, -1.0105187 , -0.45332   , ...,  1.34390459,
         0.2985838 ,  1.97058663],
       [-1.50996545,  0.98959079,  0.5096211 , ...,  0.43849455,
        -0.95368917, -0.5074631 ],
       [-0.79795355,  0.98959079,  0.38330685, ...,  0.43849455,
        -0.72867467, -0.5074631 ],
       ...,
       [-1.50996545, -1.0105187 ,  1.0148781 , ...,  0.43849455,
        -0.96159623, -0.5074631 ],
       [-1.29636188, -1.0105187 , -0.79781341, ...,  1.34390459,
        -0.93036151, -0.5074631 ],
       [ 1.55168573, -1.0105187 , -0.26138796, ..., -0.46691549,
         1.31105347,  1.97058663]])

Step #4 Train the Hierarchical Clustering Algorithm

To train a hierarchical clustering model using scikit-learn, we can use the AgglomerativeClustering or Ward class. The main parameters for these classes are:

• n_clusters: The number of clusters to form. This parameter is required for AgglomerativeClustering but is not used for Ward.
• affinity: The distance measure used to calculate the similarity between pairs of samples. This can be any of the distance measures implemented in scikit-learn, such as the Euclidean distance or the cosine similarity.
• linkage: The method used to calculate the distance between clusters. This can be one of “ward,” “complete,” “average,” or “single.”
• distance_threshold: The maximum distance between two clusters that allows them to be merged. This parameter is only used in the AgglomerativeClustering class.

To train the model, we specify the desired parameters and fit the model to the data using the fit_predict method. This method will fit the model to the data and generate predictions in one step.

# apply hierarchical clustering 
model = AgglomerativeClustering(affinity='euclidean')
predicted_segments = model.fit_predict(X_scaled)

Now we have a trained clustering model also predicted the segments for our data.

Step #5 Visualize the Results

After the model is trained, we can visualize the results to get a better understanding of the clusters that were formed. There is a wide range of plots and tools to visualize clusters. In this tutorial, we will use a scatterplot and a dendrogram.

5.1 Scatterplot

For this, we can use the lmplot function in Seaborn. The lmplot creates a 2D scatterplot with an optional overlay of a linear regression model. The plot visualizes the relationship between two variables and fits a linear regression model to the data that can highlight differences. In the following, we use this linear regression model to highlight the differences between our two cluster segments and the age of the customers.

# add predictions to data as a new column
customer_df['segment'] = predicted_segments

# create a scatter plot of the first two features, colored by segment
sns.lmplot(x="charges", y="age", hue="segment", data=customer_df, aspect=2)
plt.show()

We can see that our model has determined two clusters in our data. The clusters seem to correspond well with the smoker category, which indicates that this attribute is decisive in forming relevant groups.

5.2 Dendrogram

The hierarchical clustering approach lets us visualize relationships between different groups in our dataset in a dendrogram. A dendrogram is a graphical representation of a hierarchical structure, such as the relationships between different groups of objects or organisms. It is typically used in biology to show the relationships between different species or taxonomic groups, but it can also be used in other fields to represent the hierarchical structure of any set of data. In a dendrogram, the objects or groups being studied are represented as branches on a tree-like diagram. The branches are usually labeled with the names of the objects or groups, and the lengths of the branches represent the distances or dissimilarities between the objects or groups. The branches are also arranged in a hierarchical manner, with the most closely related objects or groups being placed closer together and the more distantly related ones being placed farther apart.

# Visualize data similarity in a dendogram
def plot_dendrogram(model, **kwargs):
    # create the counts of samples under each node
    counts = np.zeros(model.children_.shape[0])
    n_samples = len(model.labels_)
    for i, merge in enumerate(model.children_):
        current_count = 0
        for child_idx in merge:
            if child_idx < n_samples:
                current_count += 1  # leaf node
            else:
                current_count += counts[child_idx - n_samples]
        counts[i] = current_count

    linkage_matrix = np.column_stack(
        [model.children_, model.distances_, counts]
    ).astype(float)

    # Plot the corresponding dendrogram
    dendrogram(linkage_matrix, orientation='right',**kwargs)


plt.title("Hierarchical Clustering Dendrogram")
# plot the top three levels of the dendrogram
plot_dendrogram(cluster_model, truncate_mode="level", p=4)
plt.xlabel("Euclidean Distance")
plt.ylabel("Number of points in node (or index of point if no parenthesis).")
plt.show()

Source: This code block is based on code from the scikit-learn page

Summary

In conclusion, hierarchical clustering is a powerful tool for customer segmentation that can help businesses better understand their customer base and target their marketing efforts more effectively. By grouping customers into clusters based on their characteristics and behaviors, companies can create targeted campaigns and personalize their marketing efforts to better meet the needs of each group. Using Python and the scikit-learn library, we were able to apply an agglomerative clustering approach to a dataset of customer data and identify two distinct segments. We can then use these segments to inform our marketing strategies and get a better understanding of our customers.

By the way, customer segmentation is an area where real-world data can be prone to bias and unfairness. If you’re concerned about this, check out our latest article on addressing fairness in machine learning with fairlearn.

I hope this article was useful. If you have any feedback, please write your thoughts in the comments.

Sources and Further Reading

Articles

• https://scikit-learn.org/stable/auto_examples/cluster/plot_agglomerative_dendrogram.html
• Images generated with OpenAI Dall-E and Midjourney.

The links above to Amazon are affiliate links. By buying through these links, you support the Relataly.com blog and help to cover the hosting costs. Using the links does not affect the price.

Relataly articles on clustering and machine learning

• Simple Clustering using K-means in Python: This article gives an overview of cluster analysis with k-means.
• Clustering crypto markets using affinity propagation in Python: This article applies cluster analysis to crypto markets and creates a market map for various cryptocurrencies.
• Addressing fairness in machine learning with the fairlearn library

The post How to Use Hierarchical Clustering For Customer Segmentation in Python appeared first on relataly.com.

What is Facebook Prophet?

Facebook Prophet is a tool that can be used to make predictions about future events based on historical data. It was developed by Taylor and Letham, 2017, who later made it available as an open-source project. The authors developed Facebook Prophet to solve various business forecasting problems without requiring much prior knowledge. In this way, the framework addresses a significant problem many companies face today. They have various prediction problems (e.g., capacity and demand forecasting) but face a skill gap when it comes to generating reliable forecasts with techniques such as ARIMA or neural networks. Compared to that, Facebook Prophet requires minimal fine-tuning and can deal with various challenges, including seasonality, outliers, and changing trend lines. This allows Facebook Prophet to handle a wide range of forecasting problems flexibly. Before we dive into the hands-on part, let’s gain a quick overview of how Facebook Prophet works.

Also: Stock Market Prediction using Multivariate Time Series

How Facebook Prophet Works

Prerequisites

Before you proceed, ensure that you have set up your Python environment (3.8 or higher) and the required packages. If you don’t have an environment, consider following this tutorial to set up the Anaconda environment.

Also, make sure you install all required Python packages. We will be working with the following standard Python packages: 

• pandas
• seaborn
• matplotlib

In addition, we will use the Facebook Prophet library that goes by the library name “prophet.” You can install these packages using the following commands:

pip install <package name>
conda install <package name> (if you are using the anaconda packet manager)

Step #1 Loading Packages and API Key

Let’s begin by loading the required Python packages and historical price quotes for the Coca-Cola stock. We will obtain the data from the yahoo finance API. Note that the API will return several columns of data, including, opening, average, and closing prices. We will only use the closing price.

# Tested with Python 3.8.8, Matplotlib 3.5, Seaborn 0.11.1, numpy 1.19.5, plotly 4.1.1, cufflinks 0.17.3, prophet 1.1.1, CmdStan 2.31.0
import pandas as pd 
import matplotlib.pyplot as plt 
import numpy as np 
from math import log, exp 
from datetime import date, timedelta, datetime
import seaborn as sns
sns.set_style('white', {'axes.spines.right': False, 'axes.spines.top': False})
from scipy.stats import norm
from prophet import Prophet
from prophet.plot import add_changepoints_to_plot
import cmdstanpy
cmdstanpy.install_cmdstan()
cmdstanpy.install_cmdstan(compiler=True)
# Setting the timeframe for the data extraction
end_date =  date.today().strftime("%Y-%m-%d")
start_date = '2010-01-01'
# Getting quotes
stockname = 'Coca Cola'
symbol = 'KO'
# You can either use webreader or yfinance to load the data from yahoo finance
# import pandas_datareader as webreader
# df = webreader.DataReader(symbol, start=start_date, end=end_date, data_source="yahoo")
import yfinance as yf #Alternative package if webreader does not work: pip install yfinance
df = yf.download(symbol, start=start_date, end=end_date)
# Quick overview of dataset
print(df.head())

[*********************100%***********************]  1 of 1 completed
                 Open       High        Low      Close  Adj Close    Volume
Date                                                                       
2010-01-04  28.580000  28.610001  28.450001  28.520000  19.081614  13870400
2010-01-05  28.424999  28.495001  28.070000  28.174999  18.850786  23172400
2010-01-06  28.174999  28.219999  27.990000  28.165001  18.844103  19264600
2010-01-07  28.165001  28.184999  27.875000  28.094999  18.797268  13234600
2010-01-08  27.730000  27.820000  27.375000  27.575001  18.449350  28712400

Once we have downloaded the data, we create a line plot of the closing price to familiarize ourselves with the time series data. Note that Facebook Prophet works on a single input signal only (univariate data). This input will be the closing price. For illustration purposes, we add a moving average to the chart. However, the moving average makes it easier to spot trends and seasonal patterns, it will not be used to fit the model.

# Visualize the original time series
rolling_window=25
y_a_add_ma = df['Close'].rolling(window=rolling_window).mean() 
fig, ax = plt.subplots(figsize=(20,5))
sns.lineplot(data=df, x=df.index, y='Close', color='skyblue', linewidth=0.5, label='Close')
sns.lineplot(data=df, x=df.index, y=y_a_add_ma, 
    linewidth=1.0, color='royalblue', linestyle='--', label=f'{rolling_window}-Day MA')

The chart shows a long-term upward trend interrupted by phases of downturns. In addition, between 2010 and 2018, we can see some cyclical movements. At some points, we can spot clear breakpoints, for example, in 2019 and mid-2020.

Step #2 Preparing the Data

Next, we prepare our data for model training. Propjet has a strict condition on how the input columns must be named. In order to use Facebook Prophet, your data needs to be in a time series format with the time as the index and the value as the first column. In addition, column names need to adhere to the following naming convention:

• ds for the timestamp
• y for the metric columns, which in our case is the closing price

So before we proceed, we must rename the columns in our dataframe. In addition, we will remove the index and drop NA values.

df_x = df[['Close']].copy()
df_x['ds'] = df.index.copy()
df_x.rename(columns={'Close': 'y'}, inplace=True)
df_x.reset_index(inplace=True, drop=True)
df_x.dropna(inplace=True)
df_x.tail(9)

		y			ds
3257	63.139999	2022-12-09
3258	63.970001	2022-12-12
3259	63.990002	2022-12-13

Now we have a simple dataframe with ds and y as the only variables.

Step #3 Model Fitting and Forecasting

Next, let’s fit our forecasting model to the time series data. Afterward, we can make predictions about future values in the series. However, before we do this, we need to define our prediction interval.

3.1 Setting the Prediction Interval

The prediction interval is a measure of uncertainty in a forecast made with Facebook Prophet. It indicates the range within which the true value of the forecasted quantity is expected to fall a certain percentage of the time. For example, a 95% prediction interval means that the true value of the forecasted quantity is expected to fall within the given range 95% of the time.

In Facebook Prophet, the prediction interval is controlled by the interval_width parameter, which can be set when calling the predict method. The default value for interval_width is 0.80. This means that the true value of the forecasted quantity is expected to fall within the prediction interval 80% of the time. We can adjust the value of interval_width to change the width of the prediction interval as desired. In the example below, we use a prediction interval of 0.85.

3.2 Fit the Model

Next, let’s fit our model and generate a one-year forecast. First, we need to instantiate our model with by calling Prophet(). Then we use model.fit(df) to fit this model to the historical price quotes of the Coca-Cola stock. Once, we have done that, we use the model instance model.make_future_dataframe() to create an extended dataframe (future_df). This dataframe has been extended with records for a one-year period. The records are empty dummy values ready to be filled with the real forecast. We then pass this dummy dataframe to the model.predict(df) function, Facebook Prophet creates the forecast and fills up the dummy dataframe with the forecast values.

For the sake of reusability, I have encapsulated the entire process into a wrapper function. This will allow us to run quick experiments with different parameter values.

# This function fits the prophet model to the input data and generates a forecast
def fit_and_forecast(df, periods, interval_width, changepoint_range=0.8):
    # set the uncertainty interval
    Prophet(interval_width=interval_width)
    # Instantiate the model
    model = Prophet(changepoint_range=changepoint_range)
    # Fit the model
    model.fit(df)
    # Create a dataframe with a given number of dates
    future_df = model.make_future_dataframe(periods=periods)
    # Generate a forecast for the given dates
    forecast_df = model.predict(future_df)
    #print(forecast_df.head())
    return forecast_df, model, future_df
# Forecast for 365 days with full data
forecast_df, model, future_df = fit_and_forecast(df_x, 365, 0.95)
print(forecast_df.columns)
forecast_df[['yhat_lower', 'yhat_upper', 'yhat']].head(5)

Index(['ds', 'trend', 'yhat_lower', 'yhat_upper', 'trend_lower', 'trend_upper',
       'additive_terms', 'additive_terms_lower', 'additive_terms_upper',
       'weekly', 'weekly_lower', 'weekly_upper', 'yearly', 'yearly_lower',
       'yearly_upper', 'multiplicative_terms', 'multiplicative_terms_lower',
       'multiplicative_terms_upper', 'yhat'],
      dtype='object')
	yhat_lower	yhat_upper	yhat
0	24.468273	28.944286	26.691615
1	24.496074	29.146425	26.706924
2	24.513424	28.829159	26.682213
3	24.358048	28.767209	26.667476
4	24.487963	28.839966	26.666242

Voila, we have generated a one-year forecast.

Step #4 Analyzing the Forecast

Next, let’s visualize our forecast and discuss what we see. The most simple way is to create the plot with a standard Facebook Prophet function.

Also: Measuring Regression Errors with Python

model.plot(forecast_df, uncertainty=True)

So what do we see? The forecast shows that our model does not simply predict a straight line and instead has generated a more sophisticated forecast that displays an upward cyclical trend with higher highs and higher lows.

• The black dots are the data points from the historical data to which we have fit our model.
• The dark blue line is the most likely path.
• The light blue lines are the upper and lower boundaries of the prediction interval. We have set the prediction interval to 0.85, which means there is a probability of 85% the actual values will fall into this range.
• In total, the model seems confident that the price of Coca-Cola stock will rise within the next year (no financial advice). However, as we will see later, the forecast depends on where the model sees the breakpoints.

In case, you want to create a custom plot, you can use the function below.

# Visualize the Forecast
def visualize_the_forecast(df_f, df_o):
    rolling_window = 20
    yhat_mean = df_f['yhat'].rolling(window=rolling_window).mean() 
    # Thin out the ground truth data for illustration purposes
    df_lim = df_o
    # Print the Forecast
    fig, ax = plt.subplots(figsize=[20,7])
    sns.lineplot(data=df_f, x=df_f.ds, y=yhat_mean, ax=ax, label='predicted path', color='blue')
    sns.lineplot(data=df_lim, x=df_lim.ds, y='y', ax=ax, label='ground_truth', color='orange')
    #sns.lineplot(data=df_f, x=df_f.ds, y='yhat_lower', ax=ax, label='yhat_lower', color='skyblue', linewidth=1.0)
    #sns.lineplot(data=df_f, x=df_f.ds, y='yhat_upper', ax=ax, label='yhat_upper', color='coral', linewidth=1.0)
    plt.fill_between(df_f.ds, df_f.yhat_lower, df_f.yhat_upper, color='lightgreen')
    plt.legend(framealpha=0)
    ax.set(ylabel=stockname + " stock price")
    ax.set(xlabel=None)
visualize_the_forecast(forecast_df, df_x)

Step #5 Analyzing Model Components

We can gain a better understanding of different model components by using the plot_components function. This method creates a plot showing the trend, weekly and yearly seasonality, and any additional user-defined seasonalities of the forecast. This can be useful for understanding the underlying patterns in the data and for diagnosing potential issues with the model.

model.plot_components(forecast_df)

The first chart shows the trendlines that the model sees within different periods. The trendlines are separated by breakpoints about, which we will talk in the next section. When we look at the second plot, we can see no price changes during the weekend. This is plausible, considering that the stock markets are closed over the weekend. The third chart is most interesting, as it shows that the model has recognized some yearly seasonality with two peaks in April and August, as well as lows in March and October.

Step #6 Adjusting the Changepoints of our Facebook Prophet Model

Let’s take a closer look at the changepoints in our model. Changepoints are the points in time where the trend of the time series is expected to change, and Facebook Prophet’s algorithm automatically detects these points and adapts the model accordingly. Changepoints are important to Facebook Prophet because they allow the model to capture gradual changes or shifts in the data. By identifying and incorporating changepoints into the forecasting model, Facebook Prophet can make more accurate predictions. Changepoints can also help to identify potential outliers in the data.

6.1 Checking Current Changepoints

We can illustrate the changepoints in our model with the add_changepoints_to_plot method. The method adds vertical lines to a plot to indicate the locations of the changepoints in the data. By plotting the changepoints on a graph, we can visually identify when these changes in trend occur and potentially diagnose any issues with our model.

# Printing the ChangePoints of our Model
forecast_df, model, future_df = fit_and_forecast(df_x, 365, 1.0)
axislist = add_changepoints_to_plot(model.plot(forecast_df).gca(), model, forecast_df)

The chart above shows that our model has identified several changepoints in the historical data. However, it has only searched for changepoints within 80% of the time series. As a result, the algorithm hasn’t identified any change points in the most recent years after 2020. We can adjust the changepoints with the changepoint_range (default = 80%) variable. This is what we will do in the next section.

6.2 Adjusting Changepoints

We can adjust the range within which Facebook Prophet looks for changepoints with the “changepoint_range”. It is specified as a fraction of the total duration of the time series. For example, if changepoint_range is set to 0.8 and the time series spans 10 years, the algorithm will look for changepoints within the last 8 years of the series.

By default, changepoint_range is set to 0.8, which means that the algorithm will look for changepoints within the last 80% of the time series. We can adjust this value depending on the characteristics of our data and our desired level of flexibility in the model.

Increasing the value of changepoint_range will allow the algorithm to identify more changepoints and potentially improve the fit of the model, but it may also increase the risk of overfitting. Conversely, decreasing the value of changepoint_range will reduce the number of changepoints detected and may improve the model’s ability to generalize to new data, but it may also reduce the accuracy of the forecast.

Let’s fit our model again, but this time we let Facebook Prophet search for changepoints within the entire time series (changepoint_range=1.0).

# Adjusting ChangePoints of our Model
forecast_df, model, future_df = fit_and_forecast(df_x, 365, 1.0, 1.0)
axislist = add_changepoints_to_plot(model.plot(forecast_df).gca(), model, forecast_df)

The plot above shows that Facebook Prophet has now identified several additional breakpoints in the time series. As a result, the forecast has become rather pessimistic, as Facebook Prophet gave more weight to recent changes.

Finally, it is worth mentioning that it is possible to add changepoints for specific dates manually. You can try this out using “model.changepoints(series)”. The function takes a series of timestamps as the parameter value.

Summary

Sources and Further Reading

1. Taylor and Letham, 2017, Forecasting at scale
2. github.io/prophet/docs/quick_start.html
3. David Forsyth (2019) Applied Machine Learning Springer

The links above to Amazon are affiliate links. By buying through these links, you support the Relataly.com blog and help to cover the hosting costs. Using the links does not affect the price.

Other Methods for Time Series Forecasting

• Univariate time series forecasting with Recurrent Neural Networks
• Multivariate time series forecasting with Recurrent Neural Networks
• Forecasting sales data with ARIMA models

The post Univariate Stock Market Forecasting using Facebook Prophet in Python appeared first on relataly.com.

Disclaimer: This article does not constitute financial advice. Stock markets can be very volatile and are generally difficult to predict. Predictive models and other forms of analytics applied in this article only illustrate machine learning use cases.

What is OnChain Analysis?

Before discussing on-chain analysis, let’s start to recap what blockchain is. The blockchain is a decentralized distributed ledger that records transactions across a network of computers. The blockchain is composed of blocks. Each block contains a record of multiple transactions. Blocks are linked to one another, forming a chain of blocks, hence the name “blockchain.”. The blockchain is created by securely linking the blocks using cryptography, making them immutable. Each block added to the blockchain contains a cryptographic hash of the previous block, timestamp, and transaction data. In the case of Bitcoin, the data stored in the blocks include the transaction amount, the timestamp, and the unique addresses of the sender and the recipient.

Once a block has been added to the blockchain, changing the information is extremely difficult or even impossible. Moreover, unlike a normal database, the blockchain does not store its information in one place but decentrally at several participants in the network. This basic idea of decentral exchange and storage of transactions has inspired a wave of new business models and financial services that were not possible before.

On-Chain Data

On-chain data refers to data that is stored on the blockchain. It includes information such as the transaction history of a particular cryptocurrency, the balances of cryptocurrency addresses, and the smart contract code and execution history on a blockchain network. This data is stored on the blockchain and is publicly accessible to anyone with an internet connection. We can broadly classify this data into three distinct categories:

1. Transaction data (e.g., sending and receiving address, transferred amount, remaining value for a certain address)
2. Block data (e.g., timestamps, miner fees, rewards)
3. Smart contract code (i.e., codified business logic on a Blockchain)

On-chain data is an essential source of information for analysts and researchers because it provides a transparent and immutable record of activity on the blockchain. It can be used to study trends and patterns in cryptocurrency adoption and usage, as well as to track the growth and health of a blockchain network. In addition, analysts may combine on-chain data with data not stored on the blockchain. This so-called off-chain data includes, for example, price information and trading volumes.

The role of Cryptographic Proof Systems

Since the original idea, blockchain technology has evolved, and new blockchains have emerged. Changes relate in particular to the security mechanism that determines how transactions are confirmed in the network. A cryptographic proof system is a method of verifying the authenticity and integrity of data by using cryptographic techniques. Because the specific data that is stored on the blockchain may vary depending on the specific design of the blockchain and its cryptographic proof system. This means, depending on the type of blockchain, we will have different data available for our analysis.

Proof-of-work vs Proof-of-stake

The two most common consensus algorithms are proof-of-work and proof-of-stake (PoS). In the case of Bitcoin, security is guaranteed by means of the proof-of-work (PoW) procedure. In this process, so-called miners continuously spend computing power to solve cryptographic puzzles in competition with each other. The winner gets to sign a block and receives a reward for their efforts. The complexity of the puzzles is called the mining difficulty. While the mining dynamically adapts to the network’s available computing power (hash rate) and generally increases, the rewards are reduced every couple of years in a bitcoin halving event. In a PoW system, the data that is stored on the blockchain typically includes the transaction history of a particular cryptocurrency, the balances of cryptocurrency addresses, and the smart contract code and execution history on a blockchain network.

Proof of stake is an alternative to proof of work. The algorithm is designed to be more energy efficient than proof of work, as it does not require miners to perform computationally intensive work in order to create new blocks. The creator of a new block is chosen deterministically, depending on their stake in the cryptocurrency. This means that the more cryptocurrency a specific miner holds, the more likely the algorithm will enable them to create a new block. In a proof-of-stake (PoS) system, the data stored on the blockchain may include similar information, such as the transaction history and balances of cryptocurrency addresses, as well as information about the stake that is being used to secure the network.

Other types of cryptographic proof systems, such as proof-of-authority (PoA) and proof-of-elapsed-time (PoET), may store similar but not identical types of data on the blockchain.

Analyzing Blockchain Data for Bitcoin and Ethereum with Python

In this tutorial, we will explore how we can use on-chain data to gain insights into the historical development and adoption of Bitcoin and Ethereum, the two most well-known cryptocurrencies. Our analysis will focus on the adoption of the Bitcoin and Ethereum blockchains, network security, and health. By analyzing a range of data types, we can uncover interesting insights about the growth and usage of these blockchain networks. On-chain analysts use a variety of metrics to try to improve their understanding of a network and predict future price movements. The specific metrics we will be examining are:

• Metric #1 Correlation with Bitcoin Price
• Metric #2 Distribution by Holder Amount
• Metric #3 Difficulty vs. Hashrate
• Metric #4 Difficulty vs. Price
• Metric #5 Active Addresses compared to Bitcoin
• Metric #6 Transaction Count compared to Bitcoin
• Metric #7 Large Transactions compared to Bitcoin

As always, you can find the code of this tutorial on the GitHub repository.

Prerequisites

Before you proceed, ensure that you have set up your Python environment (3.8 or higher) and the required packages. If you don’t have an environment, follow this tutorial to set up the Anaconda environment. Also, make sure you install all required packages. In this tutorial, we will be working with the following standard packages: 

• pandas
• seaborn
• matplotlib

You can install packages using console commands:

pip install <package name>
conda install <package name> (if you are using the anaconda packet manager)

Obtain a CryptoCompare API Key

Accessing the CryptoCompare API requires an API key. Fortunately, there is a free tier that offers generous limits and a wide range of available data. In addition, the API has excellent documentation and offers an interactive API request builder.

You can obtain your free API Key from the CryptoCompare website by clicking “Get Your Free Key” and following the registration steps. Once you have completed the registration, you must provide your API key in any request sent to the API endpoints.

It’s a best practice not to store the key directly into your code and instead import and access the API key from a separate YAML file. Store your API key in a YAML file called “api_config_cryptocompare.yml” as follows:

api_key: “your cryptocompare api key”

Place the file into a folder from where you can import it into your Python project, e.g., “workspace/API Keys/”

Loading Packages and API Key

Let’s begin by loading the required packages and our CryptoCompare API key. The code below will load the API key from a YAML file. Should you prefer to set your key directly from the code, comment lines 18-20 and replace the “YOUR_API_KEY” with your actual API key. Make sure to keep your API key secret and secure, as it allows you to access data from the CryptoCompare API.

Note that the variables symbol_a and symbol_b define which cryptocurrencies are in the scope of the analysis. symbol_a needs to be Bitcoin because of the way how the code works. The following code sample will run the analysis for Ethereum and compare it against Bitcoin. But if you want to run the analysis for another cryptocurrency, you can change symbol_b. The prerequisite is that CryptoCompare has the respective data.

# A tutorial for this file will soon be available at www.relataly.com

# Tested with Python 3.9.13, Matplotlib 3.5.2, Seaborn 0.11.2, numpy 1.21.5, plotly 4.1.1, cryptocompare 0.7.6

import pandas as pd 
import matplotlib.pyplot as plt 
import numpy as np 
from datetime import date, timedelta, datetime
import seaborn as sns
sns.set_style('white', {'axes.spines.right': True, 'axes.spines.top': False})
import cryptocompare as cc
import requests
import IPython
import yaml
import json

# Set the API Key from a yaml file
yaml_file = open('API Keys/api_config_cryptocompare.yml', 'r')  
p = yaml.load(yaml_file, Loader=yaml.FullLoader)
api_key = p['api_key'] 
# alternatively if you have not stored your API key in a separate file
# api_key = YOUR_API_KEY

# Number of past days for which we retrieve data
data_limit = 2000

# Define coin symbols
symbol_a = 'BTC'
symbol_b = 'ETH'

We proceed by querying the CryptoCompare API to load the data for our analysis. Our data comes from three separate API endpoints:

• Historical prices for Bitcoin and Ethereum
• Onchain data for Bitcoin and Ethereum
• Bitcoin address distribution data for Bitcoin

Loading Price Data

First, we will load the price data from the cryptocompare histoday-API endpoint. This API provides us with a JSON response with a timestamp and daily prices and volume. The code below also converts the JSON response into a Pandas dataframe.

# Query price data

# Generic function for an API call to a given URL
def api_call(url):
  # Set API Key as Header
  headers = {'authorization': 'Apikey ' + api_key,}
  session = requests.Session()
  session.headers.update(headers)

  # API call to cryptocompare
  response = session.get(url)

  # Conversion of the response to dataframe
  historic_blockdata_dict = json.loads(response.text)
  df = pd.DataFrame.from_dict(historic_blockdata_dict.get('Data').get('Data'), orient='columns', dtype=None, columns=None)
  return df

def prepare_pricedata(df):
  df['date'] = pd.to_datetime(df['time'], unit='s')
  df.drop(columns=['time', 'conversionType', 'conversionSymbol'], inplace=True)
  return df

# Load the price data
base_url = 'https://min-api.cryptocompare.com/data/v2/histoday?fsym='
df_a = api_call(f'{base_url}{symbol_a}&tsym=USD&limit={data_limit}')
coin_a_price_df = prepare_pricedata(df_a)
df_b = api_call(f'{base_url}{symbol_b}&tsym=USD&limit={data_limit}')
coin_b_price_df = prepare_pricedata(df_b)
coin_b_price_df.head(3)

		high		low			open		volumefrom	volumeto		close	date
0		322.28		285.89		315.86		829138.34	2.498194e+08	292.90	2017-06-29
1		305.30		270.43		292.90		715498.52	2.054092e+08	280.68	2017-06-30
2		281.81		253.18		280.68		812033.74	2.141271e+08	261.00	2017-07-01

Now that we have the price history for Bitcoin and Ethereum, we can display the data on a line chart. Because it’s such an important event, we will also add the relevant Bitcoin halving dates. The Bitcoin halving is a built-in feature of the Bitcoin protocol that occurs approximately every four years (210,000 blocks). The purpose of the halving is to control the supply of new Bitcoins and ensure that they are released at a predictable rate. The halving reduces the reward for mining new blocks by half, which means that miners receive fewer new Bitcoins for their efforts. This helps to keep the supply of new Bitcoins in check and maintain the value of existing Bitcoins.

# Query on-chain data

# Prepare the onchain dataframe
def prepare_onchain_data(df):
  # replace the timestamp with a data and filter some faulty values
  df['date'] = pd.to_datetime(df['time'], unit='s')
  df.drop(columns='time', inplace=True)
  df = df[df['hashrate'] > 0.0]
  return df
  
base_url = 'https://min-api.cryptocompare.com/data/blockchain/histo/day?fsym='
onchain_symbol_a_df = api_call(f'{base_url}{symbol_a}&limit={data_limit}')
onchain_symbol_b_df = api_call(f'{base_url}{symbol_b}&limit={data_limit}')

# Filter some faulty values
onchain_symbol_a_df = onchain_symbol_a_df[onchain_symbol_a_df['hashrate'] > 0.0]
onchain_symbol_a_df.head(3)

	id		symbol			time		zero_balance_addresses_all_time	unique_addresses_all_time	new_addresses	active_addresses	transaction_count	transaction_count_all_time	large_transaction_count	average_transaction_value	block_height	hashrate		difficulty		block_time	block_size	current_supply
0	1182	BTC				1498694400	259466917						277866951					334750			624172				231054				235758173					10173					13.791733					473438			4.216942e+06	7.116972e+11	724.865546	966836		1.641798e+07
1	1182	BTC				1498780800	259827041						278238910					371959			727417				267360				236025533					13985					12.997582					473592			5.447359e+06	7.116972e+11	561.137255	956314		1.641990e+07
2	1182	BTC				1498867200	260153302						278544516					305606			647826				221856				236247389					10484					10.441163					473756			5.816675e+06	7.116972e+11	525.509202	882732		1.642195e+07

We can already see that the Ethereum price has been keeping up with bitcoin over the past years. Recently, the correlation has

Now that we have the price data, let’s quickly visualize it to ensure that the price charts look as expected. We will also encapsulate some of the code in helper functions. We will reuse these functions several times throughout the rest of this tutorial. For example, we will add the Bitcoin halving dates and adjust the legend to account for the two assets in the plot.

# Lineplot Helper Functions

# Adding moving averages
rolling_window = 25
coin_a_price_df['close_avg'] = coin_a_price_df['close'].rolling(window=rolling_window).mean() 
coin_b_price_df['close_avg'] = coin_b_price_df['close'].rolling(window=rolling_window).mean() 

# This function adds bitcoin halving dates as vertical lines
def add_halving_dates(ax, df_x_dates, df_ax1_y):
    halving_dates = ['2009-01-03', '2012-11-28', '2016-07-09', '2020-05-11', '2024-03-12', '2028-06-01'] 
    dates_list = [datetime.strptime(date, '%Y-%m-%d').date() for date in halving_dates]
    for i, datex in enumerate(dates_list):
        halving_ts = pd.Timestamp(datex)
        x_max = df_x_dates.max() + timedelta(days=365)
        x_min = df_x_dates.min() - timedelta(days=365)
        if (halving_ts < x_max) and (halving_ts > x_min):
            ax.axvline(x=datex, color = 'purple', linewidth=1, linestyle='dashed')
            ax.text(x=datex  + timedelta(days=20), y=df_ax1_y.max()*0.99, s='BTC Halving ' + str(i) + '\n' + str(datex), color = 'purple')

# This function creates a nice legend for twinx plots
def add_twinx_legend(ax1, ax2, x_anchor=1.18, y_anchor=1.0):
    lines_1, labels_1 = ax1.get_legend_handles_labels()
    lines_2, labels_2 = ax2.get_legend_handles_labels()
    ax1.legend(lines_1 + lines_2, labels_1 + labels_2, loc=1, facecolor='white', framealpha=0, bbox_to_anchor=(x_anchor, y_anchor))
    ax2.get_legend().remove()

# Create the lineplot
fig, ax1 = plt.subplots(figsize=(16, 6))
sns.lineplot(data=coin_a_price_df, x='date', y='close', color='cornflowerblue', linewidth=0.5, label=f'{symbol_a} close price', ax=ax1)
sns.lineplot(data=coin_a_price_df, x='date', y='close_avg', color='blue', linestyle='dashed', linewidth=1.0, 
    label=f'{symbol_a} {rolling_window}-MA', ax=ax1)
ax1.set_ylabel(f'{symbol_a} Prices')
ax1.set(xlabel=None)
ax2 = ax1.twinx()
sns.lineplot(data=coin_b_price_df, x='date', y='close', color='lightcoral', linewidth=0.5, label=f'{symbol_b} close price', ax=ax2)
sns.lineplot(data=coin_b_price_df, x='date', y='close_avg', color='red', linestyle='dashed', linewidth=1.0, 
    label=f'{symbol_b} {rolling_window}-MA', ax=ax2)
ax2.set_ylabel(f'{symbol_b} Prices')
add_twinx_legend(ax1, ax2, 0.98, 0.2)
add_halving_dates(ax1, coin_a_price_df.date, coin_a_price_df.close)
#ax1.set_yscale('log'), ax2.set_yscale('log')
plt.title(f'Prices of {symbol_a} and {symbol_b}')
plt.show()

This looks nice and proves that we have brought the data into our project and that it has a useful shape.

Loading On-Chain Data

Next, let’s load the on-chain data. To understand how a blockchain network develops and thrives, we need to look beyond price. To assess network growth, it is important to determine whether the network is being used and can increase the number of its users. Therefore, we include transaction and address data in our analysis.

We make a first API call to “data/blockchain/histo/day” to retrieve a dataset with various blockchain data. The endpoint provides daily on-chain data that includes blockchain key indicators such as:

• The number of addresses in the network
• The number of daily transactions
• Information about the blocks, incl. block size, block height, etc.
• Mining-related information, such as the mining difficulty and the available hash rate

# Prepare the onchain dataframe
def prepare_onchain_data(df):
  # replace the timestamp with a data and filter some faulty values
  df['date'] = pd.to_datetime(df['time'], unit='s')
  df.drop(columns='time', inplace=True)
  df = df[df['hashrate'] > 0.0]
  return df

# Load onchain data for Bitcoin
base_url = 'https://min-api.cryptocompare.com/data/blockchain/histo/day?fsym='
df_a = api_call(f'{base_url}{symbol_a}&limit={data_limit}')
onchain_symbol_a_df = prepare_onchain_data(df_a)

# Load onchain data for Ethereum
df_b = api_call(f'{base_url}{symbol_b}&limit={data_limit}')
onchain_symbol_b_df = prepare_onchain_data(df_b)
onchain_symbol_b_df.head(3)

	id		symbol	zero_balance_addresses_all_time	unique_addresses_all_time	new_addresses	active_addresses	transaction_count	transaction_count_all_time	large_transaction_count	average_transaction_value	block_height	hashrate	difficulty		block_time	block_size	current_supply	date
0	7605	ETH		20466340						22937123					48698			144688				259915				33294361					11528					44.835955					3950122			56.027705	962749040901496	17.183446	9460		9.289708e+07	2017-06-29
1	7605	ETH		20485843						22984680					47557			145469				249348				33543709					10791					42.018967					3955158			56.652799	972009000387636	17.157299	8800		9.292394e+07	2017-06-30
2	7605	ETH		20498357						23020671					35991			130617				235306				33779015					8715					43.389381					3960167			57.544809	992636469502805	17.249800	8105		9.295062e+07	2017-07-01

Another important indicator is how the number of coins in a cryptocurrency is distributed among the stakeholders. Unfortunately, the data required for this is not yet included in our dataset. The following code retrieves the data from a separate API endpoint (data/blockchain/balancedistribution/histo).

# Prepare balance distribution dataframe
def prepare_balancedistribution_data(df):
  df['balance_distribution'] = df['balance_distribution'].apply(lambda x: [i for i in x])
  json_struct = json.loads(df[['time','balance_distribution']].to_json(orient="records"))    
  df_ = pd.json_normalize(json_struct)
  df_['date'] = pd.to_datetime(df_['time'], unit='s')
  df_flat = pd.concat([df_.explode('balance_distribution').drop(['balance_distribution'], axis=1),
           df_.explode('balance_distribution')['balance_distribution'].apply(pd.Series)], axis=1)
  df_flat.reset_index(drop=True, inplace=True)
  df_flat['range'] = ['' + str(float(df_flat['from'][x])) + '_to_' + str(float(df_flat['to'][x])) for x in range(df_flat.shape[0])]
  df_flat.drop(columns=['from','to', 'time'], inplace=True)

  # Data cleansing
  df_flat = df_flat[~df_flat['range'].isin(['100000.0_to_0.0'])]
  df_flat['range'].iloc[df_flat['range'] == '1e-08_to_0.001'] = '0.0_to_0.001'
  return df_flat

# Load the balance distribution data for Bitcoin
base_url = 'https://min-api.cryptocompare.com/data/blockchain/balancedistribution/histo/day?fsym='
df_raw = api_call(f'{base_url}{symbol_a}&limit={data_limit}')
df_distr = prepare_balancedistribution_data(df_raw)
df_distr.head(3)

	date		totalVolume		addressesCount	range
0	2017-06-29	2068.414842		10651502.0		0.0_to_0.001
1	2017-06-29	12083.780197	3172564.0		0.001_to_0.01
2	2017-06-29	85563.613579	2753955.0		0.01_to_0.1

Now, we have all the data that we need and can proceed with our key metrics.

Metric #1 Correlation with Bitcoin Price

The first metric we will be examining is the price correlation with Bitcoin. This is an important metric to consider, as Bitcoin has a dominant position in the cryptocurrency market, and other cryptocurrencies tend to follow its price, sometimes with larger fluctuations. During bull markets, when Bitcoin reaches new highs, other cryptocurrencies tend to see strong price performance. Conversely, during bear markets, when Bitcoin experiences prolonged price declines, most other cryptocurrencies tend to underperform. There are occasional deviations from this pattern, which are usually related to economic or technical changes on the respective networks. The rolling price correlation helps us to understand these types of developments better.

To illustrate how the correlation between the two cryptocurrencies has evolved, we calculate rolling correlations. This means we are applying a correlation between the two time series of Bitcoin and Ethereum as a rolling window calculation. We define 100 days as the window for each calculation.

# Calculate the Rolling Correlation Coefficient
rolling_window = 100 #days

# Generate a work dataframe that includes closing prices and date
df_price_merged = pd.DataFrame.from_dict(data={f'close_{symbol_b}': coin_b_price_df['close'], f'close_{symbol_a}': coin_a_price_df['close'], 'date': coin_a_price_df['date']})
# Create the rolling correlation dataframe
df_temp = pd.DataFrame({'cor': coin_b_price_df.close.rolling(rolling_window).corr(coin_a_price_df.close).dropna()})
# Reverse the index and join the df to create a date index
df_cor_dateindex = df_price_merged.join(df_temp[::-1].set_index(df_temp.index)).dropna().set_index('date')

# Create the plot
fig, ax1 = plt.subplots(figsize=(16, 6))
label = f'{symbol_a}-{symbol_b} correlation (rolling window={rolling_window})'
sns.lineplot(data=df_cor_dateindex, x=df_cor_dateindex.index, y='cor', color='royalblue', linewidth=1.0, label=label)
add_halving_dates(ax1, df_cor_dateindex.index, df_cor_dateindex[f'cor'])
plt.legend(framealpha=0)
plt.title(label)

The chart shows that the correlation between Bitcoin and Ethereum has been in the range between 0.95 and -0.2 for quite some time. Currently, both cryptocurrencies are heavily correlated.

Metric #2 Distribution by Holder Amount

Another important aspect to consider is the distribution of coin value among the players in the network. If the majority of coins are concentrated in the hands of a few players, this can pose a risk to the price. This is especially true for proof-of-value networks like Ethereum, where the number of coins owned by players in the network affects their importance to the network. In addition, the distribution of coins can provide insight into price movements. For example, an increase in the number of addresses with a disproportionately large number of coins may be interpreted as a bullish sign, indicating that large players with significant market power are becoming optimistic. On the other hand, a decrease in the number of large addresses may be seen as a bearish sign.

The following code block will display the historical distribution of coins in the Bitcoin network. The data includes the number of addresses in the network that hold a specific amount of Bitcoins, and it distinguishes between different address sizes (e.g., “0.001 – 0.01 BTC”, “0.01 – 0.1 BTC”, and “0.1 – 1 BTC”). We will specifically look at the growth rates in the different holding ranges. A rising line thus means that the growth of th number of addresses in this range accelerates.

# Prepare address distribution data for plotting
df_distr_add = df_distr.copy()
for i in list(df_distr_add.range.unique()):
    df_distr_add.loc[df_distr.range == i, 'addressesCount_pct_cum'] = df_distr_add[df_distr_add.range == i]['addressesCount'].pct_change().dropna().cumsum().rolling(window=50).mean()
df_distr_add.dropna(inplace=True)
# Lineplot: Address Count by Holder Amount
fig, ax1 = plt.subplots(figsize=(16, 6))
sns.lineplot(data=df_distr_add, x='date', hue='range', linewidth = 1.0, y='addressesCount_pct_cum', ax=ax1, palette='bright')
plt.ylabel('Percentage Growth')
ax1.tick_params(axis="x", rotation=90, labelsize=10, length=0)
ax1.set(xlabel=None)
plt.title(f'Percentage Growth in the Distribution of Total Address Count for {symbol_a} by Holder Amount')
plt.legend(framealpha=0)
plt.show()

There are a couple of things to denote:

• We can see that the number of large Bitcoin addresses (yellow line) has recently declined (negative growth) but is currently increasing again. This may be a sign that large whales are accumulating Bitcoins again.
• We can also see that the growth rates of smaller addresses are accelerating (orange, green, red, purple lines), which means that the holdings get spread across a wider network.

Metric #3 Difficulty vs. Hashrate

The distribution of coins within a blockchain network is an important factor to consider. The hash rate measures the total computing power available on the network, and a higher hash rate makes it more difficult for attackers to launch a 51% attack, leading to increased network security. The mining difficulty determines how hard it is to mine the next block, and it is measured by the number of hashes that must be generated to find a valid solution.

The difficulty is adjusted periodically to ensure that new blocks are added to the blockchain at a consistent rate. If the hash rate of the network increases significantly, the difficulty will also increase to compensate. This helps to ensure that the rate at which new blocks are added to the blockchain remains constant, regardless of changes in the hash rate.

# Lineplot: Difficulty vs Hashrate
fig, ax1 = plt.subplots(figsize=(16, 6))
sns.lineplot(data=onchain_symbol_a_df, x='date', y='difficulty', 
    linewidth=1.0, color='royalblue', ax=ax1, label=f'{symbol_a} mining difficulty')
ax2 = ax1.twinx()
sns.lineplot(data=onchain_symbol_a_df[::5], x='date', y='hashrate', 
    linewidth=1.0, color='red', ax=ax2, label=f'{symbol_a} network hashrate')
add_twinx_legend(ax1, ax2, 0.98, 0.2)
add_halving_dates(ax1, onchain_symbol_a_df.date, onchain_symbol_a_df.difficulty)
ax1.set(xlabel=None)
plt.title(f'{symbol_a} Mining Difficulty vs Hashrate')
plt.show()

Hash rate and mining difficulty are closely related, which results from the fact that mining difficulty is adjusted periodically. However, it is essential to note that the hash rate does not show the distribution of the computing power in the network. A high hash rate alone does not guarantee network security if it is provided by a small number of parties. To assess the security of a proof-of-work blockchain, we therefore must also look at how the hash rate is distributed.

Metric #4 Difficulty vs. Price

Next, we will compare the difficulty vs. Price. The price of Bitcoin is an essential indicator of the demand for cryptocurrency. When the price is high, it can attract more miners to the network, as they are motivated by the potential to earn a high return on their investment. This can lead to an increase in the overall hash rate of the network, which makes it more secure against attacks. On the other hand, when the price is low, it may discourage miners from joining the network, leading to a decrease in the hash rate and potentially making the network more vulnerable to attacks.

# Add a moving average
rolling_window = 25
coin_a_price_df['close_avg'] = coin_a_price_df['close'].rolling(window=rolling_window).mean() 
# Creating a Lineplot: Mining Difficulty vs Price
fig, ax1 = plt.subplots(figsize=(16, 6))
sns.lineplot(data=onchain_symbol_a_df, x='date', y='difficulty', linewidth=1.0, color='orangered', ax=ax1, label=f'mining difficulty')
ax2 = ax1.twinx()
sns.lineplot(data=coin_a_price_df, x='date', y='close', linewidth=0.5, color='skyblue', ax=ax2, label=f'close price')
sns.lineplot(data=coin_a_price_df, x='date', y='close_avg', linewidth=1.0, linestyle='--', color='royalblue', ax=ax2, label=f'MA-100')
add_twinx_legend(ax1, ax2, 0.98, 0.2)
add_halving_dates(ax1, onchain_symbol_a_df.date, onchain_symbol_a_df.hashrate)
ax1.set(xlabel=None)
ax1.set(ylabel='Mining Difficulty')
plt.title(f'{symbol_a} Mining Difficulty vs Price')
plt.show()

Currently, the price of Bitcoin is low while the mining difficulty is high. As a result, it may be less attractive for miners to join the network. This is because the low price means that the potential reward for mining a new block may not be as high as it could be if the price were higher. At the same time, the high difficulty means that it will be more challenging for miners to find a valid solution to the mathematical problems they are working on, which could lead to lower profits.

In this situation, the overall hash rate of the network may decrease, as some miners may choose to leave the network or scale back their mining operations. This could make the network more vulnerable to attacks, as a lower hash rate means that there is less computing power available to secure the network.

Metric #5 Active Addresses compared to Bitcoin

Next, let’s compare the number of active addresses between Ethereum and Bitcoin. An active address in a blockchain is a unique address that has conducted a transaction within a certain time period. The number of active addresses on a blockchain can be a useful metric for analyzing the usage and adoption of the network. There are several reasons why active addresses are important:

• Network usage: The number of active addresses can give an indication of how much the network is being used. A higher number of active addresses may suggest that more people are using the network to send and receive transactions.
• Network growth: An increase in the number of active addresses over time may indicate that the network is growing and attracting more users. This could be a positive sign for the long-term health and sustainability of the network.
• Network health: The number of active addresses may also provide insight into the overall health of the network. For example, a sudden drop in the number of active addresses could be a sign of trouble, such as a loss of user confidence or a technical issue.
• Network security: The number of active addresses can also be used as a rough proxy for the level of decentralization on the network. A large and diverse set of active addresses may indicate that the network is decentralized and less vulnerable to attacks.

# Calculate active addresses moving average
rolling_window=25
y_a_add_ma = onchain_symbol_a_df['active_addresses'].rolling(window=rolling_window).mean() 
y_b_add_ma = onchain_symbol_b_df['active_addresses'].rolling(window=rolling_window).mean() 

# Lineplot: Active Addresses
fig, ax1 = plt.subplots(figsize=(16, 6))
sns.lineplot(data=onchain_symbol_a_df[-1*data_limit::10], x='date', y='active_addresses', 
    linewidth=0.5, color='skyblue', ax=ax1, label=f'{symbol_a} active addresses')
sns.lineplot(data=onchain_symbol_a_df[-1*data_limit::10], x='date', y=y_a_add_ma, 
    linewidth=1.0, color='royalblue', linestyle='--', ax=ax1, label=f'{symbol_a} active addresses {rolling_window}-Day MA')
sns.lineplot(data=onchain_symbol_b_df[-1*data_limit::10], x='date', y='active_addresses', 
    linewidth=0.5, color='lightcoral', ax=ax1, label=f'{symbol_b} active addresses')
sns.lineplot(data=onchain_symbol_b_df[-1*data_limit::10], x='date', y=y_b_add_ma, 
    linewidth=1.0, color='red', linestyle='--', ax=ax1, label=f'{symbol_b} active addresses {rolling_window}-Day MA')
ax1.set(xlabel=None)
ax1.set(ylabel='Active Addresses')
plt.title(f'Active Addresses: {symbol_b} vs {symbol_a}')
plt.legend(framealpha=0)
plt.show()

Metric #6 Transaction Count compared to Bitcoin

Transaction count is an important metric for analyzing the usage and adoption of a blockchain. It refers to the total number of transactions that have been processed on the network over a given time period.

There are several reasons why transaction count is important:

• Network usage: The transaction count can give an indication of network usage. A higher transaction count may suggest that more people are using the network to send and receive transactions.
• Network growth: An increase in the transaction count over time may indicate that the network is growing and attracting more users. This could be a positive sign for the long-term health and sustainability of the network.
• Network health: The transaction count may also provide insight into the overall health of the network. For example, a sudden drop in the transaction count could be a sign of trouble, such as a loss of user confidence or a technical issue.
• Network security: The transaction count can be used as a rough proxy for the level of decentralization on the network. A large and diverse set of transactions may indicate that the network is decentralized and less vulnerable to attacks.

# Calculate Transaction Count Moving Averages
rolling_window=25
y_a_trx_ma = onchain_symbol_a_df['transaction_count'].rolling(window=rolling_window).mean() 
y_b_trx_ma = onchain_symbol_b_df['transaction_count'].rolling(window=rolling_window).mean() 

# Lineplot: Transactions Count
fig, ax1 = plt.subplots(figsize=(16, 6))
sns.lineplot(data=onchain_symbol_a_df[-1*data_limit::10], x='date', y='transaction_count', 
    linewidth=0.5, color='skyblue', ax=ax1, label=f'{symbol_a} transactions')
sns.lineplot(data=onchain_symbol_a_df[-1*data_limit::10], x='date', y=y_a_trx_ma, 
    linewidth=1.0, color='royalblue', linestyle='--', ax=ax1, label=f'{symbol_a} transactions {rolling_window}-Day MA')
sns.lineplot(data=onchain_symbol_b_df[-1*data_limit::10], x='date', y='transaction_count', 
    linewidth=0.5, color='lightcoral', ax=ax1, label=f'{symbol_b} transactions')
sns.lineplot(data=onchain_symbol_b_df[-1*data_limit::10], x='date', y=y_b_trx_ma, 
    linewidth=1.0, color='red', linestyle='--', ax=ax1, label=f'{symbol_b} transactions {rolling_window}-Day MA')
ax1.set(xlabel=None)
ax1.set(ylabel='Transaction Count')
plt.legend(framealpha=0)
plt.title(f'Transactions: {symbol_b} vs {symbol_a}')
plt.show()

As the first blockchain, Bitcoin has always been the most crucial cryptocurrency in the crypto space. However, there are now blockchains based on more modern methods. Recently, the crypto community has been discussing whether Ethereum is about to overtake Bitcoin. But how is the situation in terms of transactions?

The chart shows that the use of blockchains has changed throughout the last few years. Ethereum has seen strong growth in the number of transactions, while the number of Bitcoin transactions has stagnated for some time.

Metric #7 Large Transactions compared to Bitcoin

Another metric to look at is the number of large transactions. Large transactions on a blockchain, also known as “whale transactions,” refer to transactions involving a significant amount of cryptocurrency. These transactions may be important to analyze for a number of reasons:

1. Market impact: Large transactions can have a significant impact on the market, as they involve a large amount of cryptocurrency being bought or sold. This can affect the supply and demand for the cryptocurrency and potentially impact its price.
2. Network security: Large transactions may also be of interest from a security standpoint, as they may be more likely to attract the attention of attackers. If a large transaction is successfully compromised, it could have a significant impact on the network.
3. Network health: Large transactions may provide insight into the overall health of the network. For example, a sudden increase in large transactions may indicate an increased demand for cryptocurrency, while a decrease in large transactions could be a sign of trouble.
4. Network usage: Large transactions can also be an indicator of how the network is being used. For example, a high number of large transactions may suggest that the network is being used for high-value transactions, while a low number may suggest that it is being used for smaller, everyday transactions.

# Calculate Large Transactions Moving Averages
rolling_window=25
y_a_ltrx_ma = onchain_symbol_a_df['large_transaction_count'].rolling(window=rolling_window).mean() 
y_b_ltrx_ma = onchain_symbol_b_df['large_transaction_count'].rolling(window=rolling_window).mean() 
# Lineplot: Large Transactions
fig, ax1 = plt.subplots(figsize=(16, 6))
sns.lineplot(data=onchain_symbol_a_df[-1*data_limit::10], x='date', y='large_transaction_count', 
    linewidth=0.5, color='skyblue', ax=ax1, label=f'{symbol_a} large transactions')
sns.lineplot(data=onchain_symbol_a_df[-1*data_limit::10], x='date', y=y_a_ltrx_ma, 
    linewidth=1.0, color='royalblue', linestyle='--', ax=ax1, label=f'{symbol_a} large transactions MA-{window}')
sns.lineplot(data=onchain_symbol_b_df[-1*data_limit::10], x='date', y='large_transaction_count', 
    linewidth=0.5, color='lightcoral', ax=ax1, label=f'{symbol_b} large transactions')
sns.lineplot(data=onchain_symbol_b_df[-1*data_limit::10], x='date', y=y_b_ltrx_ma, 
    linewidth=1.0, color='red', linestyle='--', ax=ax1, label=f'{symbol_b} large transaction MA-{window}')
ax1.set(ylabel='Large Transactions')
plt.title(f'Large Transactions > 100k: {symbol_b} vs {symbol_a}')
plt.legend(framealpha=0)
plt.show()

As we can see, both networks have recently experienced a decline in the number of large transactions.

Summary

Bitcoin and blockchain technology are transforming the financial sector and have seen increasing adoption during the past decade. Due to the increasing need to better understand complex blockchain networks, the importance of on-chain analytics is growing. This article has demonstrated how we can analyze blockchain data with Python. We used the CryptoCompare API to query various On-Chain and Off-Chain data for Bitcoin and Ethereum. By combining blockchain with these data, we gained several important insights into what has been happening in the crypto space over the past few years. Among other things, we have

• …compared the historical evolution of mining difficulty and network hash rate.
• …analyzed the usage of the Ethereum and Bitcoin blockchains.
• …and highlighted how the distribution of Bitcoin holdings has evolved in the past years.

Our analysis in this article focused solely on Bitcoin and Ethereum. However, you can easily analyze other blockchains by replacing the symbols used in the API calls.

I hope you liked this post, and I would appreciate your feedback. Is OnChain analysis a topic that you want to see covered more often? Or do you want to see more articles on deep learning and machine learning? Let me know in the comments.

Sources and Further Reading

1. Antony Lewis (2018) Basics of Bitcoins and Blockchains
2. CryptoCompare API
3. glassnode.com
4. OpenAI ChatGPT was carefully used to revise certain parts of this article

The links above to Amazon are affiliate links. By buying through these links, you support the Relataly.com blog and help to cover the hosting costs. Using the links does not affect the price.

The post On-Chain Analytics: Metrics for Analyzing Blockchains in Python appeared first on relataly.com.

What is Pandas Data Reader?

The Pandas DataReader library provides functions that extract data from various Internet sources into a pandas DataFrame. The pandas DataReader supports several remote data providers, including Alpha Vantage, World Bank, Eurostat, the OECD, and several stock markets such as the NASDAQ, Yahoo Finance, and Naver Finance. A complete list of available sources is available from the pandas DataReader API documentation.

Access Financial Data using Pandas DataReader and the Yahoo Finance REST API in Python

In this tutorial, we will learn how to use the pandas library to retrieve data for the German stock market index DAX from the Yahoo finance API. Specifically, we will use the pandas_datareader package, which provides a convenient interface for accessing various online data sources. We will carry out the following steps:

1. Install the pandas_datareader package.
2. Import the necessary libraries in our Python script.
3. Use the data.DataReader function to request data for the DAX index from the Yahoo finance API. Specify the start and end dates for the data you want to retrieve. The returned data will be stored in a pandas DataFrame.
4. Finally, we use the plot() method from the matplotlib library to visualize the data.

The code is available on the GitHub repository.

View on GitHub Relataly GitHub Repo

Prerequisites

pip install <package name>
conda install <package name> (if you are using the anaconda packet manager)

pip install pandas-datareader

Step #1: Define the API Request Parameters

We begin by setting up imports and adjusting the request parameters. The parameters in an API request will depend on the API and the library used for making the request. Also, some parameters may be optional, while others are mandatory.

The Yahoo Finance API allows us to limit the period we want to retrieve price data, an example of an optional parameter. Furthermore, we need to define the ticker symbol for the financial instrument if we wish to request the price data. This parameter is mandatory.

The ticker symbol for the German stock market index is ^GDAXI. If you want to retrieve price data for other stocks or indices, you can search for the respective ticker symbols on Yahoo finance.

import pandas_datareader as webreader
import pandas as pd
import matplotlib.pyplot as plt

# Set the API 
data_source = "yahoo"

# Set the API parameters
date_today = "2020-01-01" # period start date
date_start = "2010-01-01" # period end date
symbol = "^GDAXI" # asset symbol - For more symbols check yahoo.finance.com

Step #2: Send the Request to the REST API Endpoint

Once we have defined the request parameters, we can make the request via the DataReader function and print out the result. If you request a REST API, the response will come back in JSON format. However, DataReader will directly convert the API response into a DataFrame, which makes using APIs much simpler.

This will retrieve the DAX stock market index from Yahoo Finance and print the first few rows of the resulting DataFrame. We have specified a date range to retrieve data for a specific period of time.

# Send the request to the yahoo finance api endpoint
df = webreader.DataReader(symbol, start=date_start, end=date_today, data_source=data_source)
df.head(5)

			High		Low			Open		Close		Volume		Adj Close
Date						
2010-01-04	6048.299805	5974.430176	5975.520020	6048.299805	104344400.0	6048.299805
2010-01-05	6058.020020	6015.669922	6043.939941	6031.859863	117572100.0	6031.859863
2010-01-06	6047.569824	5997.089844	6032.390137	6034.330078	108742400.0	6034.330078
2010-01-07	6037.569824	5961.250000	6016.799805	6019.359863	133704300.0	6019.359863
2010-01-08	6053.040039	5972.240234	6028.620117	6037.609863	126099000.0	6037.609863

Dataframe with the price data from yahoo finance.

Step #3 Plot the Data

Let’s quickly print the data to check if everything looks ok.

# Plot the closing prices
fig, ax1 = plt.subplots(figsize=(12, 8))
plt.plot(df.index, df.Close)
plt.show()

Everything looks good, so let’s proceed.

Step #4: Save the Data to a CSV File

To save the data from a Pandas DataFrame to a CSV file, you can use the to_csv method. The to_csv method takes a few optional arguments that you can use to customize the output. For example, you can use the “sep” argument to specify a different delimiter to use in the CSV file or the “index” argument for including or excluding the DataFrame’s index in the output.

Here’s an example of how you can use the to_csv method with the index parameter set to False:

# Save the data to a CSV file
df.to_csv("price_quotes.csv", index=False)

Now you have the data on your local machine and can load it later. So unless you require more actual data, there is no need to call the API again.

Summary

This article has shown how to use the Pandas DataReader library. We learned how to use the library to request data from the Yahoo Finance API and save the data to a Pandas DataFrame. The Pandas DataReader library is a helpful tool for importing financial data into a Pandas DataFrame and working with it in Python. You can use it to retrieve data from a wide range of sources, including stock prices from major stock exchanges, economic data from the Federal Reserve, and cryptocurrency prices. Once you have the data in a DataFrame, you can use the various methods and functions provided by Pandas to analyze and manipulate the data, and save the results to a CSV file using the to_csv method.

I hope this post was helpful. If you have any remarks or questions, let me know.

Sources and Further Reading

pandas-datareader.readthedocs.io/

Images created with Midjourney AI

Further API Tutorials

• 

  Generating Detailed Images with OpenAI DALL-E and ChatGPT in Python: A Step-By-Step API Tutorial

• 

  Unleashing the Power of ChatGPT and Other OpenAI GPT Language Models in Python A Guide to Using APIs

• 

  Univariate Stock Market Forecasting using Facebook Prophet in Python

• 

  On-Chain Analytics: Metrics for Analyzing Blockchains in Python

• 

  Requesting Crypto Price Data from the Gate.io REST API in Python

• 

  Posting Tweets On Twitter using Python and Tweepy

The post Using Pandas DataReader to Access Online Data Sources in Python appeared first on relataly.com.