Preamble

You may be missing the imblearn package below. With anaconda, you can grab it using conda install -c conda-forge imbalanced-learn

import numpy as np
import pandas as pd
import plotly.graph_objects as go
from imblearn.over_sampling import ADASYN

Introduction

We've covered class imbalance and over sampling in another section, so this will just serve as another example on a different dataset. In this section, we'll be using the Cardiotocography (CTG) dataset located at https://archive.ics.uci.edu/ml/datasets/cardiotocography. It has 23 attributes, 2 of which are two different classifications of the same samples, CLASS (1 to 10) and NSP (1 to 3).

Downloading the Dataset

To keep this notebook independant, we will download the CTG dataset within our code. If you've already downloaded it and viewed it using some spreadsheet software, you will have noticed that we want to use the "Data" spreadsheet, and that we also want to drop he first row.

url = (
    "https://archive.ics.uci.edu/ml/machine-learning-databases/00193/CTG.xls"
)

data = pd.read_excel(url, sheet_name="Data", header=1)

Let's have a quick look at what we've downloaded.

data.head()

Preparing the Dataset

Looking at the data, we can see the last 3 rows are not samples

data.tail()

So we will get rid of them.

data = data.drop(data.tail(3).index)

Let's check to make sure they're gone.

data.tail()

Now let's create a dataframe that only contains our input features and our desired classification. In the spreadsheet we know our 21 features were labelled with the numbers 1 to 21, starting from the column LB and ending on the column Tendency. Let's display all the columns in our dataframe.

data.columns

Index(['b', 'e', 'AC', 'FM', 'UC', 'DL', 'DS', 'DP', 'DR', 'Unnamed: 9', 'LB',
       'AC.1', 'FM.1', 'UC.1', 'DL.1', 'DS.1', 'DP.1', 'ASTV', 'MSTV', 'ALTV',
       'MLTV', 'Width', 'Min', 'Max', 'Nmax', 'Nzeros', 'Mode', 'Mean',
       'Median', 'Variance', 'Tendency', 'Unnamed: 31', 'A', 'B', 'C', 'D',
       'E', 'AD', 'DE', 'LD', 'FS', 'SUSP', 'Unnamed: 42', 'CLASS',
       'Unnamed: 44', 'NSP'],
      dtype='object')

There are many ways to reduce all of these columns down to the 21 we want. In this notebook, we're going to be explicit with our column selections to ensure no errors can be introduced from a change in the external spreadsheet data.

columns = [
    "LB",
    "AC.1",
    "FM.1",
    "UC.1",
    "DL.1",
    "DS.1",
    "DP.1",
    "ASTV",
    "MSTV",
    "ALTV",
    "MLTV",
    "Width",
    "Min",
    "Max",
    "Nmax",
    "Nzeros",
    "Mode",
    "Mean",
    "Median",
    "Variance",
    "Tendency",
]

features = data[columns]

Let's have a peek at our new dataframe.

features.head()

Let's also create a new dataframe to store our labels, the NSP column, which contains our desired classifications.

labels = data["NSP"]

Again, we'll check to see what we've created.

labels.head()

0    2.0
1    1.0
2    1.0
3    1.0
4    1.0
Name: NSP, dtype: float64

Frequency of each Classication

Now let's see how many samples we have for each class.

value_counts = labels.value_counts()

fig = go.Figure()
fig.add_bar(x=value_counts.index, y=value_counts.values)
fig.show()

We can see that our dataset is imbalanced, with class 1 having a significantly higher number of samples. This may be a problem for us depending on our approach, so we may want to balance this dataset before continuing our study.

Oversampling

We could perform over-sampling using an implementation of the Adaptive Synthetic (ADASYN) sampling approach in the imbalanced-learn library. We will pass in our features and labels variables, and expect outputs that have been resampled for balanced frequency.

features_resampled, labels_resampled = ADASYN().fit_resample(features, labels)

Let's see how many samples we have for each class this time.

labels_resampled = pd.DataFrame(labels_resampled, columns=["NSP"])
value_counts = labels_resampled["NSP"].value_counts()

fig = go.Figure()
fig.add_bar(x=value_counts.index, y=value_counts.values)
fig.show()

Much better!

Conclusion

In this section, we've used adaptive synthetic sampling to resample and balance our CTG dataset. The output is a balanced dataset, however, it's important to remember that these approaches should only be applied to training data, and never to data that is to be used for testing. For your own experiments, make sure you only apply an approach like this after you have already split your dataset and locked away the test set for later.