US Mortality - Race and Manner of Death

Dr. Shahin Rostami

2020-06-09

Preamble¶

In [1]:

import numpy as np                   # for multi-dimensional containers 
import pandas as pd                  # for DataFrames
import itertools
from chord import Chord

Introduction¶

In previous sections, we visualised co-occurrences of Pokémon type. Whilst it was interesting to look at, the dataset only contained Pokémon from the first six geerations. In this section, we're going to use the TidyTuesday Animal Crossing villagers dataset to visualise the relationship between Species and .

The Dataset¶

The dataset documentation states that we can expect 13 variables per each of the 1017 Pokémon of the first eight generations.

Let's download the mirrored dataset and have a look for ourselves.

In [2]:

data_url = '/Users/shahin/Documents/devel/data/2015_data.csv'
data = pd.read_csv(data_url)
data.head()

/Users/shahin/opt/miniconda3/envs/analytics/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3063: DtypeWarning: Columns (40,41,42,43,61,62,63,64) have mixed types.Specify dtype option on import or set low_memory=False.
  interactivity=interactivity, compiler=compiler, result=result)

Out[2]:

	resident_status	education_1989_revision	education_2003_revision	education_reporting_flag	month_of_death	sex	detail_age_type	detail_age	age_substitution_flag	age_recode_52	...	record_condition_18	record_condition_19	record_condition_20	race	bridged_race_flag	race_imputation_flag	race_recode_3	race_recode_5	hispanic_origin	hispanic_originrace_recode
0	1	NaN	3.0	1	1	M	1	84	NaN	42	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6
1	1	NaN	6.0	1	1	M	1	70	NaN	40	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6
2	1	NaN	3.0	1	1	F	1	91	NaN	44	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6
3	1	NaN	3.0	1	1	F	1	40	NaN	34	...	NaN	NaN	NaN	3	NaN	NaN	2	3	100	8
4	1	NaN	5.0	1	1	F	1	89	NaN	43	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6

5 rows × 77 columns

In [3]:

data['race_recode_5'].value_counts()

Out[3]:

1    2311103
2     320759
4      67295
3      19041
Name: race_recode_5, dtype: int64

In [4]:

data.columns

Out[4]:

Index(['resident_status', 'education_1989_revision', 'education_2003_revision',
       'education_reporting_flag', 'month_of_death', 'sex', 'detail_age_type',
       'detail_age', 'age_substitution_flag', 'age_recode_52', 'age_recode_27',
       'age_recode_12', 'infant_age_recode_22',
       'place_of_death_and_decedents_status', 'marital_status',
       'day_of_week_of_death', 'current_data_year', 'injury_at_work',
       'manner_of_death', 'method_of_disposition', 'autopsy', 'activity_code',
       'place_of_injury_for_causes_w00_y34_except_y06_and_y07_',
       'icd_code_10th_revision', '358_cause_recode', '113_cause_recode',
       '130_infant_cause_recode', '39_cause_recode',
       'number_of_entity_axis_conditions', 'entity_condition_1',
       'entity_condition_2', 'entity_condition_3', 'entity_condition_4',
       'entity_condition_5', 'entity_condition_6', 'entity_condition_7',
       'entity_condition_8', 'entity_condition_9', 'entity_condition_10',
       'entity_condition_11', 'entity_condition_12', 'entity_condition_13',
       'entity_condition_14', 'entity_condition_15', 'entity_condition_16',
       'entity_condition_17', 'entity_condition_18', 'entity_condition_19',
       'entity_condition_20', 'number_of_record_axis_conditions',
       'record_condition_1', 'record_condition_2', 'record_condition_3',
       'record_condition_4', 'record_condition_5', 'record_condition_6',
       'record_condition_7', 'record_condition_8', 'record_condition_9',
       'record_condition_10', 'record_condition_11', 'record_condition_12',
       'record_condition_13', 'record_condition_14', 'record_condition_15',
       'record_condition_16', 'record_condition_17', 'record_condition_18',
       'record_condition_19', 'record_condition_20', 'race',
       'bridged_race_flag', 'race_imputation_flag', 'race_recode_3',
       'race_recode_5', 'hispanic_origin', 'hispanic_originrace_recode'],
      dtype='object')

capitalise the name, personality, and species of each villager.

In [5]:

#data['manner'] = data['manner_of_death']#.str.capitalize()
#data['race_recode_5'] = data['race_recode_5']#.str.capitalize()
#data['species'] = data['species'].str.capitalize()

It looks good so far, but let's confirm the 13 variables against 1017 samples from the documentation.

In [6]:

data.shape

Out[6]:

(2718198, 77)

Perfect, that's exactly what we were expecting.

Data Wrangling¶

We need to do a bit of data wrangling before we can visualise our data. We can see from the columns names that the Pokémon types are split between the columns Type 1 and Type 2.

In [7]:

pd.DataFrame(data.columns.values.tolist())

Out[7]:

	0
0	resident_status
1	education_1989_revision
2	education_2003_revision
3	education_reporting_flag
4	month_of_death
...	...
72	race_imputation_flag
73	race_recode_3
74	race_recode_5
75	hispanic_origin
76	hispanic_originrace_recode

77 rows × 1 columns

So let's select just these two columns and work with a list containing only them as we move forward.

In [8]:

data.fillna(0, inplace=True)

In [ ]:

In [9]:

data.iloc[6572].manner_of_death

Out[9]:

7.0

In [11]:

data.manner_of_death.value_counts()

Out[11]:

7.0    2107352
0.0     388364
1.0     143961
2.0      44417
3.0      18885
5.0      11054
4.0       4165
Name: manner_of_death, dtype: int64

In [ ]:

{'143961': 'Accident',
 '44417': 'Suicide',
 '18885': 'Homicide',
 '4165': 'Pending investigation',
 '11054': 'Could not determine',
 '2107352': 'Natural',
 '388364': 'Not specified'}

In [10]:

import json

In [11]:

with open("/Users/shahin/Documents/devel/data/2015_data.json", "r") as read_file:
    codes = json.load(read_file)

In [12]:

codes['manner_of_death']

Out[12]:

{'1': 'Accident',
 '2': 'Suicide',
 '3': 'Homicide',
 '4': 'Pending investigation',
 '5': 'Could not determine',
 '7': 'Natural',
 'Blank': 'Not specified'}

In [13]:

codes['manner_of_death']['0'] = codes['manner_of_death'].pop('Blank')

In [14]:

remove = ["Natural", "Not specified", "Could not determine", "Pending investigation"]

In [15]:

list(codes['manner_of_death'].values())

Out[15]:

['Accident',
 'Suicide',
 'Homicide',
 'Pending investigation',
 'Could not determine',
 'Natural',
 'Not specified']

In [16]:

left = list(codes['race_recode_5'].values())
pd.DataFrame(left)

Out[16]:

	0
0	White
1	Black
2	American Indian
3	Asian or Pacific Islander

In [17]:

right = list(codes['manner_of_death'].values())
pd.DataFrame(right)

Out[17]:

	0
0	Accident
1	Suicide
2	Homicide
3	Pending investigation
4	Could not determine
5	Natural
6	Not specified

In [18]:

right = [x for x in right if x not in remove]

In [19]:

data['manner_of_death'] = data['manner_of_death'].astype('int32')
data['manner_of_death'] = data['manner_of_death'].astype('str')

data.iloc[6572].manner_of_death

Out[19]:

'7'

In [20]:

left.sort()
right.sort()

In [21]:

left

Out[21]:

['American Indian', 'Asian or Pacific Islander', 'Black', 'White']

In [22]:

data = data.replace({"manner_of_death": codes['manner_of_death']})

In [23]:

data['race_recode_5'] = data['race_recode_5'].astype('int32')
data['race_recode_5'] = data['race_recode_5'].astype('str')

data.iloc[6572].race_recode_5

Out[23]:

'2'

In [24]:

data = data.replace({"race_recode_5": codes['race_recode_5']})

In [25]:

data.iloc[6572].race_recode_5

Out[25]:

'Black'

In [26]:

manner_race = pd.DataFrame(data[['manner_of_death', 'race_recode_5']].values)
manner_race

Out[26]:

	0	1
0	Natural	White
1	Natural	White
2	Natural	White
3	Homicide	American Indian
4	Natural	White
...	...	...
2718193	Natural	Black
2718194	Natural	White
2718195	Natural	White
2718196	Natural	Black
2718197	Natural	Black

2718198 rows × 2 columns

Now for the names of our types.

Which we can now use to create the matrix.

In [27]:

features= left+right
d = pd.DataFrame(0, index=features, columns=features)

Our chord diagram will need two inputs: the co-occurrence matrix, and a list of names to label the segments.

We can build a co-occurrence matrix with the following approach. We'll start by creating a list with every type pairing in its original and reversed form.

In [28]:

manner_race = list(itertools.chain.from_iterable((i, i[::-1]) for i in manner_race.values))

In [29]:

for x in manner_race:
    if(x[0] not in remove and x[1] not in remove):
        d.at[x[0], x[1]] += 1

In [30]:

Out[30]:

	American Indian	Asian or Pacific Islander	Black	White	Accident	Homicide	Suicide
American Indian	0	0	0	0	2067	304	582
Asian or Pacific Islander	0	0	0	0	3032	364	1334
Black	0	0	0	0	15746	9419	2518
White	0	0	0	0	123116	8798	39983
Accident	2067	3032	15746	123116	0	0	0
Homicide	304	364	9419	8798	0	0	0
Suicide	582	1334	2518	39983	0	0	0

In [31]:

for race in left:
    d.loc[ race , : ] = ((d.loc[ race , : ] / d.loc[ race , : ].sum()) * 100)
    d.loc[ : , race  ] = ((d.loc[  : , race  ] / d.loc[  : , race  ].sum()) * 100)

In [32]:

(d[race].value_counts(normalize=True)*100).astype(int)

Out[32]:

0.000000     57
71.621960    14
23.259859    14
5.118181     14
Name: White, dtype: int64

In [33]:

Out[33]:

	American Indian	Asian or Pacific Islander	Black	White	Accident	Homicide	Suicide
American Indian	0.000000	0.00000	0.000000	0.000000	69.996614	10.294616	19.708771
Asian or Pacific Islander	0.000000	0.00000	0.000000	0.000000	64.101480	7.695560	28.202960
Black	0.000000	0.00000	0.000000	0.000000	56.879673	34.024492	9.095835
White	0.000000	0.00000	0.000000	0.000000	71.621960	5.118181	23.259859
Accident	69.996614	64.10148	56.879673	71.621960	0.000000	0.000000	0.000000
Homicide	10.294616	7.69556	34.024492	5.118181	0.000000	0.000000	0.000000
Suicide	19.708771	28.20296	9.095835	23.259859	0.000000	0.000000	0.000000

Chord Diagram¶

Time to visualise the co-occurrence of types using a chord diagram. We are going to use a list of custom colours that represent the types.

In [34]:

colors = ["#2DE1FC","#1883B4","#C5DB66","#90B64D","#DB2B39","#E76926", "#DB9118"]

In [35]:

names = left + right

Finally, we can put it all together.

In [ ]:

In [36]:

names[1]= "Asian or PI"


names[0]= "AIAN"

In [37]:

Out[37]:

	American Indian	Asian or Pacific Islander	Black	White	Accident	Homicide	Suicide
American Indian	0.000000	0.00000	0.000000	0.000000	69.996614	10.294616	19.708771
Asian or Pacific Islander	0.000000	0.00000	0.000000	0.000000	64.101480	7.695560	28.202960
Black	0.000000	0.00000	0.000000	0.000000	56.879673	34.024492	9.095835
White	0.000000	0.00000	0.000000	0.000000	71.621960	5.118181	23.259859
Accident	69.996614	64.10148	56.879673	71.621960	0.000000	0.000000	0.000000
Homicide	10.294616	7.69556	34.024492	5.118181	0.000000	0.000000	0.000000
Suicide	19.708771	28.20296	9.095835	23.259859	0.000000	0.000000	0.000000

Finally, we can put it all together but this time with the details matrix passed in.

In [38]:

Chord(
    d.round(2).values.tolist(),
    names,
    colors=colors,
    credit=True,
      wrap_labels=True,
      margin=50, 
    font_size_large=7,
divide=True,
    noun="percent",
    divide_idx=len(left),
    divide_size=.2,
    width=850).show()

Chord Diagram

Chord Diagram with Names¶

It would be nice to show a list of Pokémon names when hovering over co-occurring Pokémon types. To do this, we can make use of the optional details parameter.

In [42]:

Out[42]:

	American Indian	Asian or Pacific Islander	Black	White	Accident	Homicide	Suicide
American Indian	0.000000	0.00000	0.000000	0.000000	69.996614	10.294616	19.708771
Asian or Pacific Islander	0.000000	0.00000	0.000000	0.000000	64.101480	7.695560	28.202960
Black	0.000000	0.00000	0.000000	0.000000	56.879673	34.024492	9.095835
White	0.000000	0.00000	0.000000	0.000000	71.621960	5.118181	23.259859
Accident	69.996614	64.10148	56.879673	71.621960	0.000000	0.000000	0.000000
Homicide	10.294616	7.69556	34.024492	5.118181	0.000000	0.000000	0.000000
Suicide	19.708771	28.20296	9.095835	23.259859	0.000000	0.000000	0.000000

Next, we'll create an empty multi-dimensional array with the same shape as our matrix.

In [39]:

details = np.empty((len(names),len(names)),dtype=object)
details_thumbs = np.empty((len(names),len(names)),dtype=object)

Now we can populate the details array with lists of Pokémon names in the correct positions.

In [40]:

for count_x, item_x in enumerate(names):
    for count_y, item_y in enumerate(names):
        details_urls = data[
            (data['species'].isin([item_x, item_y])) &
            (data['personality'].isin([item_y, item_x]))]['url'].to_list()
        
        details_names = data[
            (data['species'].isin([item_x, item_y])) &
            (data['personality'].isin([item_y, item_x]))]['name'].to_list()
        
        urls_names = np.column_stack((details_urls, details_names))
        if(urls_names.size > 0):
            details[count_x][count_y] = details_names
            details_thumbs[count_x][count_y] = details_urls

        else:
            details[count_x][count_y] = []
            details_thumbs[count_x][count_y] = []

details=pd.DataFrame(details).values.tolist()
details_thumbs=pd.DataFrame(details_thumbs).values.tolist()

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/opt/miniconda3/envs/analytics/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2645             try:
-> 2646                 return self._engine.get_loc(key)
   2647             except KeyError:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'species'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-40-f54e3a0b2906> in <module>
      2     for count_y, item_y in enumerate(names):
      3         details_urls = data[
----> 4             (data['species'].isin([item_x, item_y])) &
      5             (data['personality'].isin([item_y, item_x]))]['url'].to_list()
      6 

~/opt/miniconda3/envs/analytics/lib/python3.7/site-packages/pandas/core/frame.py in __getitem__(self, key)
   2798             if self.columns.nlevels > 1:
   2799                 return self._getitem_multilevel(key)
-> 2800             indexer = self.columns.get_loc(key)
   2801             if is_integer(indexer):
   2802                 indexer = [indexer]

~/opt/miniconda3/envs/analytics/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2646                 return self._engine.get_loc(key)
   2647             except KeyError:
-> 2648                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   2649         indexer = self.get_indexer([key], method=method, tolerance=tolerance)
   2650         if indexer.ndim > 1 or indexer.size > 1:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'species'

In [ ]:

len(right)

In [ ]:

Chord(d.values.tolist(), names,credit=True, colors=colors, wrap_labels=False,
      margin=40, font_size_large=7,details=details,details_thumbs=details_thumbs,noun="villagers",
        details_separator="", divide=True, divide_idx=len(left),divide_size=.2, width=850).show()

In [ ]:

np.empty(shape=(6,1)).tolist()

Conclusion¶

In this section, we demonstrated how to conduct some data wrangling on a downloaded dataset to prepare it for a chord diagram. Our chord diagram is interactive, so you can use your mouse or touchscreen to investigate the co-occurrences!

Visualisation of Co-occurring Types

Dr. Shahin Rostami

2020-05-31

Preamble¶

In [2]:

:dep darn = {version = "0.3.0"}
:dep ndarray = {version = "0.13.1"}
:dep itertools = {version = "0.9.0"}
:dep chord = {Version = "0.1.6"}
extern crate ndarray;

use ndarray::prelude::*;
use itertools::Itertools;
use chord::{Chord, Plot};

Introduction¶

In this section, we're going to use the Complete Pokemon Dataset dataset to visualise the co-occurrence of Pokémon types from generations one to eight. We'll make this happen using a chord diagram.

Chord Diagrams¶

In a chord diagram (or radial network), entities are arranged radially as segments with their relationships visualised by arcs that connect them. The size of the segments illustrates the numerical proportions, whilst the size of the arc illustrates the significance of the relationships¹.

Chord diagrams can be useful when trying to convey relationships between different entities, and they can be beautiful and eye-catching. They can get messy when considering many entities, so it's often beneficial to make them interactive and explorable.

The Chord Crate¶

I wasn't able to find any Rust crates for plotting chord diagrams, so I ported my own (based on d3-chord) from Python to Rust.

You can get the crate either from crates.io or from the GitHub repository. With your processed data, you should be able to plot something beautiful with just a single line, Chord{ matrix : matrix, names : names, .. Chord::default() }.show(). To enable the pro features of the chord crate check out Chord Pro.

The Dataset¶

The dataset documentation states that we can expect two type variables per each of the 1028 samples of the first eight generations, type_1, and type_2.

Let's download the mirrored dataset and have a look for ourselves.

In [3]:

let data = darn::read_csv("https://datacrayon.com/datasets/pokemon_gen_1_to_8.csv");

In [4]:

darn::show_frame(&data.0, Some(&data.1));

Out[4]:

	pokedex_number	name	german_name	japanese_name	generation	status	species	type_number	type_1	type_2	height_m	weight_kg	abilities_number	ability_1	ability_2	ability_hidden	total_points	hp	attack	defense	sp_attack	sp_defense	speed	catch_rate	base_friendship	base_experience	growth_rate	egg_type_number	egg_type_1	egg_type_2	percentage_male	egg_cycles	against_normal	against_fire	against_water	against_electric	against_grass	against_ice	against_fight	against_poison	against_ground	against_flying	against_psychic	against_bug	against_rock	against_ghost	against_dragon	against_dark	against_steel	against_fairy
"0"	"1"	"Bulbasaur"	"Bisasam"	"フシギダネ (Fushigidane)"	"1"	"Normal"	"Seed Pokémon"	"2"	"Grass"	"Poison"	"0.7"	"6.9"	"2"	"Overgrow"	""	"Chlorophyll"	"318"	"45"	"49"	"49"	"65"	"65"	"45"	"45"	"70"	"64"	"Medium Slow"	"2"	"Grass"	"Monster"	"87.5"	"20"	"1"	"2"	"0.5"	"0.5"	"0.25"	"2"	"0.5"	"1"	"1"	"2"	"2"	"1"	"1"	"1"	"1"	"1"	"1"	"0.5"
"1"	"2"	"Ivysaur"	"Bisaknosp"	"フシギソウ (Fushigisou)"	"1"	"Normal"	"Seed Pokémon"	"2"	"Grass"	"Poison"	"1"	"13"	"2"	"Overgrow"	""	"Chlorophyll"	"405"	"60"	"62"	"63"	"80"	"80"	"60"	"45"	"70"	"142"	"Medium Slow"	"2"	"Grass"	"Monster"	"87.5"	"20"	"1"	"2"	"0.5"	"0.5"	"0.25"	"2"	"0.5"	"1"	"1"	"2"	"2"	"1"	"1"	"1"	"1"	"1"	"1"	"0.5"
"2"	"3"	"Venusaur"	"Bisaflor"	"フシギバナ (Fushigibana)"	"1"	"Normal"	"Seed Pokémon"	"2"	"Grass"	"Poison"	"2"	"100"	"2"	"Overgrow"	""	"Chlorophyll"	"525"	"80"	"82"	"83"	"100"	"100"	"80"	"45"	"70"	"236"	"Medium Slow"	"2"	"Grass"	"Monster"	"87.5"	"20"	"1"	"2"	"0.5"	"0.5"	"0.25"	"2"	"0.5"	"1"	"1"	"2"	"2"	"1"	"1"	"1"	"1"	"1"	"1"	"0.5"
"3"	"3"	"Mega Venusaur"	"Bisaflor"	"フシギバナ (Fushigibana)"	"1"	"Normal"	"Seed Pokémon"	"2"	"Grass"	"Poison"	"2.4"	"155.5"	"1"	"Thick Fat"	""	""	"625"	"80"	"100"	"123"	"122"	"120"	"80"	"45"	"70"	"281"	"Medium Slow"	"2"	"Grass"	"Monster"	"87.5"	"20"	"1"	"1"	"0.5"	"0.5"	"0.25"	"1"	"0.5"	"1"	"1"	"2"	"2"	"1"	"1"	"1"	"1"	"1"	"1"	"0.5"
"4"	"4"	"Charmander"	"Glumanda"	"ヒトカゲ (Hitokage)"	"1"	"Normal"	"Lizard Pokémon"	"1"	"Fire"	""	"0.6"	"8.5"	"2"	"Blaze"	""	"Solar Power"	"309"	"39"	"52"	"43"	"60"	"50"	"65"	"45"	"70"	"62"	"Medium Slow"	"2"	"Dragon"	"Monster"	"87.5"	"20"	"1"	"0.5"	"2"	"1"	"0.5"	"0.5"	"1"	"1"	"2"	"1"	"1"	"0.5"	"2"	"1"	"1"	"1"	"0.5"	"0.5"
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
"1023"	"888"	"Zacian Hero of Many Battles"	""	""	"8"	"Legendary"	"Warrior Pokémon"	"1"	"Fairy"	""	"2.8"	"110"	"1"	"Intrepid Sword"	""	""	"670"	"92"	"130"	"115"	"80"	"115"	"138"	""	""	""	"Slow"	"1"	"Undiscovered"	""	""	"120"	"1"	"1"	"1"	"1"	"1"	"1"	"0.5"	"2"	"1"	"1"	"1"	"0.5"	"1"	"1"	"0"	"0.5"	"2"	"1"
"1024"	"889"	"Zamazenta Crowned Shield"	""	""	"8"	"Legendary"	"Warrior Pokémon"	"2"	"Fighting"	"Steel"	"2.9"	"785"	"1"	"Dauntless Shield"	""	""	"720"	"92"	"130"	"145"	"80"	"145"	"128"	""	""	""	"Slow"	"1"	"Undiscovered"	""	""	"120"	"0.5"	"2"	"1"	"1"	"0.5"	"0.5"	"2"	"0"	"2"	"1"	"1"	"0.25"	"0.25"	"1"	"0.5"	"0.5"	"0.5"	"1"
"1025"	"889"	"Zamazenta Hero of Many Battles"	""	""	"8"	"Legendary"	"Warrior Pokémon"	"1"	"Fighting"	""	"2.9"	"210"	"1"	"Dauntless Shield"	""	""	"670"	"92"	"130"	"115"	"80"	"115"	"138"	""	""	""	"Slow"	"1"	"Undiscovered"	""	""	"120"	"1"	"1"	"1"	"1"	"1"	"1"	"1"	"1"	"1"	"2"	"2"	"0.5"	"0.5"	"1"	"1"	"0.5"	"1"	"2"
"1026"	"890"	"Eternatus"	""	""	"8"	"Legendary"	"Gigantic Pokémon"	"2"	"Poison"	"Dragon"	"20"	"950"	"1"	"Pressure"	""	""	"690"	"140"	"85"	"95"	"145"	"95"	"130"	""	""	""	"Slow"	"1"	"Undiscovered"	""	""	"120"	"1"	"0.5"	"0.5"	"0.5"	"0.25"	"2"	"0.5"	"0.5"	"2"	"1"	"2"	"0.5"	"1"	"1"	"2"	"1"	"1"	"1"
"1027"	"890"	"Eternatus Eternamax"	""	""	"8"	"Legendary"	"Gigantic Pokémon"	"2"	"Poison"	"Dragon"	"100"	""	"0"	""	""	""	"1125"	"255"	"115"	"250"	"125"	"250"	"130"	""	""	""	"Slow"	"1"	"Undiscovered"	""	""	"120"	"1"	"0.5"	"0.5"	"0.5"	"0.25"	"2"	"0.5"	"0.5"	"2"	"1"	"2"	"0.5"	"1"	"1"	"2"	"1"	"1"	"1"

It looks good so far, we can clearly see the two type columns. Let's confirm that we have 1028 samples.

In [5]:

&data.0.shape()

Out[5]:

[1028, 51]

Perfect, that's exactly what we were expecting.

Data Wrangling¶

We need to do a bit of data wrangling before we can visualise our data. We can see from the column names that the Pokémon types are split between the columns type_1 and type_2.

In [6]:

&data.1

Out[6]:

["", "pokedex_number", "name", "german_name", "japanese_name", "generation", "status", "species", "type_number", "type_1", "type_2", "height_m", "weight_kg", "abilities_number", "ability_1", "ability_2", "ability_hidden", "total_points", "hp", "attack", "defense", "sp_attack", "sp_defense", "speed", "catch_rate", "base_friendship", "base_experience", "growth_rate", "egg_type_number", "egg_type_1", "egg_type_2", "percentage_male", "egg_cycles", "against_normal", "against_fire", "against_water", "against_electric", "against_grass", "against_ice", "against_fight", "against_poison", "against_ground", "against_flying", "against_psychic", "against_bug", "against_rock", "against_ghost", "against_dragon", "against_dark", "against_steel", "against_fairy"]

So let's select just these two columns and work with a list containing only them as we move forward.

In [7]:

let types = data.0.slice(s![.., 9..11]).into_owned();
darn::show_frame(&types, None);

Out[7]:

"Grass"	"Poison"
"Grass"	"Poison"
"Grass"	"Poison"
"Grass"	"Poison"
"Fire"	""
...	...
"Fairy"	""
"Fighting"	"Steel"
"Fighting"	""
"Poison"	"Dragon"
"Poison"	"Dragon"

Our chord diagram will need two inputs: the co-occurrence matrix, and a list of names to label the segments.

First, we'll populate our list of type names by looking for the unique ones.

In [8]:

let mut names = types.iter().cloned().unique().collect_vec();
names

Out[8]:

["Grass", "Poison", "Fire", "", "Flying", "Dragon", "Water", "Bug", "Normal", "Dark", "Electric", "Psychic", "Ground", "Ice", "Steel", "Fairy", "Fighting", "Rock", "Ghost"]

Let's sort this alphabetically.

In [9]:

names.sort();
names

Out[9]:

["", "Bug", "Dark", "Dragon", "Electric", "Fairy", "Fighting", "Fire", "Flying", "Ghost", "Grass", "Ground", "Ice", "Normal", "Poison", "Psychic", "Rock", "Steel", "Water"]

We'll also remove the empty string that has appeared as a result of samples with only one type.

In [10]:

names.remove(0);
names

Out[10]:

["Bug", "Dark", "Dragon", "Electric", "Fairy", "Fighting", "Fire", "Flying", "Ghost", "Grass", "Ground", "Ice", "Normal", "Poison", "Psychic", "Rock", "Steel", "Water"]

Now we can create our empty co-occurrence matrix with a shape that can hold co-occurrences between our types.

In [11]:

let type_count = names.len();
let mut matrix: Vec<Vec<f64>> = vec![vec![Default::default(); type_count]; type_count];
matrix

Out[11]:

[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]]

We can populate a co-occurrence matrix with the following approach. Here, we're looping through every sample in our dataset and incrementing the corresponding matrix entry by one using the type_1 and type_2 indices from the names vector. To make sure we have a co-occurrence matrix, we're also doing the same in reverse, i.e. type_2 and type_1.

In [12]:

for item in types.genrows() { 
    if(!item[0].is_empty() && !item[1].is_empty()) {
        matrix[names.iter().position(|s| s == &item[1]).unwrap()]
              [names.iter().position(|s| s == &item[0]).unwrap()] += 1.0;
        matrix[names.iter().position(|s| s == &item[0]).unwrap()]
              [names.iter().position(|s| s == &item[1]).unwrap()] += 1.0;
    };
};

Chord Diagram¶

Time to visualise the co-occurrence of types using a chord diagram. We are going to use a list of custom colours that represent the types.

In [13]:

let colors: Vec<String> = vec![
    "#A6B91A", "#705746", "#6F35FC", "#F7D02C", "#D685AD",
    "#C22E28", "#EE8130", "#A98FF3", "#735797", "#7AC74C",
    "#E2BF65", "#96D9D6", "#A8A77A", "#A33EA1", "#F95587",
    "#B6A136", "#B7B7CE", "#6390F0"
]
.into_iter()
.map(String::from)
.collect();

Finally, we can put it all together.

In [14]:

Chord {
    matrix: matrix.clone(),
    names: names.clone(),
    colors: colors,
    margin: 30.0,
    wrap_labels: true,
    ..Chord::default()
}
.show();

Out[14]:

Chord Diagram

Conclusion¶

Co-occurring Pokémon Types

Dr. Shahin Rostami

2020-05-18

Preamble¶

In [2]:

import numpy as np                   # for multi-dimensional containers 
import pandas as pd                  # for DataFrames
import itertools
from chord import Chord

Introduction¶

In previous sections, we visualised co-occurrences of Pokémon type. Whilst it was interesting to look at, the dataset only contained Pokémon from the first six geerations. In this section, we're going to use the Pokemon with stats Generation 8 dataset to visualise the co-occurrence of Pokémon types from generations one to eight.

The Dataset¶

The dataset documentation states that we can expect 13 variables per each of the 1017 Pokémon of the first eight generations.

Let's download the mirrored dataset and have a look for ourselves.

In [3]:

data_url = 'https://datacrayon.com/datasets/pokemon_gen_1_to_8.csv'
data = pd.read_csv(data_url)
data.head()

Out[3]:

	Unnamed: 0	pokedex_number	name	german_name	japanese_name	generation	status	species	type_number	type_1	...	against_ground	against_flying	against_psychic	against_bug	against_rock	against_ghost	against_dragon	against_dark	against_steel	against_fairy
0	0	1	Bulbasaur	Bisasam	フシギダネ (Fushigidane)	1	Normal	Seed Pokémon	2	Grass	...	1.0	2.0	2.0	1.0	1.0	1.0	1.0	1.0	1.0	0.5
1	1	2	Ivysaur	Bisaknosp	フシギソウ (Fushigisou)	1	Normal	Seed Pokémon	2	Grass	...	1.0	2.0	2.0	1.0	1.0	1.0	1.0	1.0	1.0	0.5
2	2	3	Venusaur	Bisaflor	フシギバナ (Fushigibana)	1	Normal	Seed Pokémon	2	Grass	...	1.0	2.0	2.0	1.0	1.0	1.0	1.0	1.0	1.0	0.5
3	3	3	Mega Venusaur	Bisaflor	フシギバナ (Fushigibana)	1	Normal	Seed Pokémon	2	Grass	...	1.0	2.0	2.0	1.0	1.0	1.0	1.0	1.0	1.0	0.5
4	4	4	Charmander	Glumanda	ヒトカゲ (Hitokage)	1	Normal	Lizard Pokémon	1	Fire	...	2.0	1.0	1.0	0.5	2.0	1.0	1.0	1.0	0.5	0.5

5 rows × 51 columns

It looks good so far, but let's confirm the 13 variables against 1017 samples from the documentation.

In [4]:

data.shape

Out[4]:

(1028, 51)

Perfect, that's exactly what we were expecting.

Data Wrangling¶

We need to do a bit of data wrangling before we can visualise our data. We can see from the columns names that the Pokémon types are split between the columns Type 1 and Type 2.

In [5]:

pd.DataFrame(data.columns.values.tolist())

Out[5]:

	0
0	Unnamed: 0
1	pokedex_number
2	name
3	german_name
4	japanese_name
5	generation
6	status
7	species
8	type_number
9	type_1
10	type_2
11	height_m
12	weight_kg
13	abilities_number
14	ability_1
15	ability_2
16	ability_hidden
17	total_points
18	hp
19	attack
20	defense
21	sp_attack
22	sp_defense
23	speed
24	catch_rate
25	base_friendship
26	base_experience
27	growth_rate
28	egg_type_number
29	egg_type_1
30	egg_type_2
31	percentage_male
32	egg_cycles
33	against_normal
34	against_fire
35	against_water
36	against_electric
37	against_grass
38	against_ice
39	against_fight
40	against_poison
41	against_ground
42	against_flying
43	against_psychic
44	against_bug
45	against_rock
46	against_ghost
47	against_dragon
48	against_dark
49	against_steel
50	against_fairy

So let's select just these two columns and work with a list containing only them as we move forward.

In [6]:

types = pd.DataFrame(data[['type_1', 'type_2']].values)
types

Out[6]:

	0	1
0	Grass	Poison
1	Grass	Poison
2	Grass	Poison
3	Grass	Poison
4	Fire	NaN
...	...	...
1023	Fairy	NaN
1024	Fighting	Steel
1025	Fighting	NaN
1026	Poison	Dragon
1027	Poison	Dragon

1028 rows × 2 columns

Without further investigation, we can see that we have at least a few NaN values in the table above. We are only interested in co-occurrence of types, so we can remove all samples which contain a NaN value.

In [7]:

types = types.dropna()

We can also see an instance where the type Fighting at index $1014$ is followed by \n. We'll strip all these out before continuing.

In [8]:

types = types.replace('\n','', regex=True)
types

Out[8]:

	0	1
0	Grass	Poison
1	Grass	Poison
2	Grass	Poison
3	Grass	Poison
6	Fire	Flying
...	...	...
1021	Dragon	Ghost
1022	Fairy	Steel
1024	Fighting	Steel
1026	Poison	Dragon
1027	Poison	Dragon

542 rows × 2 columns

Our chord diagram will need two inputs: the co-occurrence matrix, and a list of names to label the segments.

First we'll populate our list of type names by looking for the unique ones.

In [9]:

names = np.unique(types).tolist()
pd.DataFrame(names)

Out[9]:

	0
0	Bug
1	Dark
2	Dragon
3	Electric
4	Fairy
5	Fighting
6	Fire
7	Flying
8	Ghost
9	Grass
10	Ground
11	Ice
12	Normal
13	Poison
14	Psychic
15	Rock
16	Steel
17	Water

Now we can create our empty co-occurrence matrix using these type names for the row and column indeces.

In [10]:

matrix = pd.DataFrame(0, index=names, columns=names)
matrix

Out[10]:

	Bug	Dark	Dragon	Electric	Fairy	Fighting	Fire	Flying	Ghost	Grass	Ground	Ice	Normal	Poison	Psychic	Rock	Steel	Water
Bug	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Dark	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Dragon	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Electric	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Fairy	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Fighting	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Fire	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Flying	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Ghost	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Grass	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Ground	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Ice	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Normal	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Poison	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Psychic	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Rock	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Steel	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Water	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0

We can populate a co-occurrence matrix with the following approach. We'll start by creating a list with every type pairing in its original and reversed form.

In [11]:

types = list(itertools.chain.from_iterable((i, i[::-1]) for i in types.values))

Which we can now use to create the matrix.

In [12]:

for t in types:
    matrix.at[t[0], t[1]] += 1
    
matrix = matrix.values.tolist()

We can list the DataFrame for better presentation.

In [13]:

pd.DataFrame(matrix)

Out[13]:

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
0	0	0	0	4	2	4	4	14	1	6	2	2	0	13	2	5	7	5
1	0	0	4	2	3	3	4	5	3	3	3	2	5	5	3	2	2	7
2	0	4	0	3	1	2	3	8	5	5	9	3	1	4	5	2	2	3
3	4	2	3	0	2	0	1	6	1	1	1	2	2	3	1	3	4	3
4	2	3	1	2	0	0	0	2	1	5	0	1	5	1	9	3	5	4
5	4	3	2	0	0	0	7	1	1	3	0	1	4	2	6	1	4	3
6	4	4	3	1	0	7	0	7	5	0	4	1	2	2	3	3	1	1
7	14	5	8	6	2	1	7	0	3	7	4	2	27	3	7	6	3	8
8	1	3	5	1	1	1	5	3	0	12	6	1	0	4	3	0	4	2
9	6	3	5	1	5	3	0	7	12	0	1	3	2	15	3	2	3	3
10	2	3	9	1	0	0	4	4	6	1	0	3	1	2	2	9	6	10
11	2	2	3	2	1	1	1	2	1	3	3	0	0	0	4	2	2	7
12	0	5	1	2	5	4	2	27	0	2	1	0	0	0	5	0	0	1
13	13	5	4	3	1	2	2	3	4	15	2	0	0	0	0	1	0	6
14	2	3	5	1	9	6	3	7	3	3	2	4	5	0	0	2	9	6
15	5	2	2	3	3	1	3	6	0	2	9	2	0	1	2	0	7	11
16	7	2	2	4	5	4	1	3	4	3	6	2	0	0	9	7	0	1
17	5	7	3	3	4	3	1	8	2	3	10	7	1	6	6	11	1	0

Chord Diagram¶

Time to visualise the co-occurrence of types using a chord diagram. We are going to use a list of custom colours that represent the types.

In [14]:

colors = ["#A6B91A", "#705746", "#6F35FC", "#F7D02C", "#D685AD",
          "#C22E28", "#EE8130", "#A98FF3", "#735797", "#7AC74C",
          "#E2BF65", "#96D9D6", "#A8A77A", "#A33EA1", "#F95587",
          "#B6A136", "#B7B7CE", "#6390F0"]

In [15]:

names

Out[15]:

['Bug',
 'Dark',
 'Dragon',
 'Electric',
 'Fairy',
 'Fighting',
 'Fire',
 'Flying',
 'Ghost',
 'Grass',
 'Ground',
 'Ice',
 'Normal',
 'Poison',
 'Psychic',
 'Rock',
 'Steel',
 'Water']

Finally, we can put it all together.

In [16]:

Chord(matrix, names, colors=colors).show()

Chord Diagram

Chord Diagram with Names¶

It would be nice to show a list of Pokémon names and images when hovering over co-occurring Pokémon types. To do this, we can make use of the optional details parameter.

Let's clean up our dataset by removing all instances of \n.

In [17]:

data = data.replace('\n','', regex=True)

Let's also add a column to our dataset to store URLs that point to the images.

In [18]:

data['URL'] = ""

for index, row in data.iterrows():
    dex = f"{row['pokedex_number']:03d}"
    url = f"https://shahinrostami.com/images/data-is-beautiful/pokemon_thumbs/{dex}.png"
    data.at[index,'URL'] = url

Next, we'll create an empty multi-dimensional arrays with the same shape as our matrix for our details and thumbnail images.

In [19]:

details = np.empty((len(names),len(names)),dtype=object)
details_thumbs = np.empty((len(names),len(names)),dtype=object)

Now we can populate the details array with lists of Pokémon names in the correct positions.

In [20]:

for count_x, item_x in enumerate(names):
    for count_y, item_y in enumerate(names):
        details_urls = data[
            (data['type_1'].isin([item_x, item_y])) &
            (data['type_2'].isin([item_y, item_x]))]['URL'].to_list()
        
        details_names = data[
            (data['type_1'].isin([item_x, item_y])) &
            (data['type_2'].isin([item_y, item_x]))]['name'].to_list()
        
        urls_names = np.column_stack((details_urls, details_names))
        if(urls_names.size > 0):
            details[count_x][count_y] = details_names
            details_thumbs[count_x][count_y] = details_urls

        else:
            details[count_x][count_y] = []
            details_thumbs[count_x][count_y] = []

details=pd.DataFrame(details).values.tolist()
details_thumbs=pd.DataFrame(details_thumbs).values.tolist()

Finally, we can put it all together but this time with the details matrix passed in.

In [27]:

Chord(
    matrix,
    names,
    colors=colors,
    details=details,
    details_thumbs=details_thumbs,
    margin=80,
    noun="Pokémon",
    thumbs_width=70,
    thumbs_margin=1,
    popup_width=600,
    thumbs_font_size=10,
    credit=True
).show()

Chord Diagram

Conclusion¶

Animal Crossing Villagers - Species and Personalities

Dr. Shahin Rostami

2020-05-17

Preamble¶

In [1]:

import numpy as np                   # for multi-dimensional containers 
import pandas as pd                  # for DataFrames
import itertools
from chord import Chord

Introduction¶

The Dataset¶

The dataset documentation states that we can expect 13 variables per each of the 1017 Pokémon of the first eight generations.

Let's download the mirrored dataset and have a look for ourselves.

In [2]:

data_url = 'https://shahinrostami.com/datasets/ac_villagers.csv'
data = pd.read_csv(data_url)
data.head()

Out[2]:

	id	name	gender	species	birthday	personality	song	phrase	full_id	url
0	admiral	Admiral	male	bird	Jan-27	cranky	Steep Hill	aye aye	villager-admiral	https://shahinrostami.com/images/data-is-beaut...
1	agent-s	Agent S	female	squirrel	07-Feb	peppy	DJ K.K.	sidekick	villager-agent-s	https://shahinrostami.com/images/data-is-beaut...
2	agnes	Agnes	female	pig	Apr-21	uchi	K.K. House	snuffle	villager-agnes	https://shahinrostami.com/images/data-is-beaut...
3	al	Al	male	gorilla	Oct-18	lazy	Steep Hill	Ayyeeee	villager-al	https://shahinrostami.com/images/data-is-beaut...
4	alfonso	Alfonso	male	alligator	06-Sep	lazy	Forest Life	it'sa me	villager-alfonso	https://shahinrostami.com/images/data-is-beaut...

capitalise the name, personality, and species of each villager.

In [3]:

data['name'] = data['name'].str.capitalize()
data['personality'] = data['personality'].str.capitalize()
data['species'] = data['species'].str.capitalize()

It looks good so far, but let's confirm the 13 variables against 1017 samples from the documentation.

In [4]:

data.shape

Out[4]:

(391, 10)

Perfect, that's exactly what we were expecting.

Data Wrangling¶

We need to do a bit of data wrangling before we can visualise our data. We can see from the columns names that the Pokémon types are split between the columns Type 1 and Type 2.

In [5]:

pd.DataFrame(data.columns.values.tolist())

Out[5]:

	0
0	id
1	name
2	gender
3	species
4	birthday
5	personality
6	song
7	phrase
8	full_id
9	url

So let's select just these two columns and work with a list containing only them as we move forward.

In [6]:

species_personality = pd.DataFrame(data[['species', 'personality']].values)
species_personality

Out[6]:

	0	1
0	Bird	Cranky
1	Squirrel	Peppy
2	Pig	Uchi
3	Gorilla	Lazy
4	Alligator	Lazy
...	...	...
386	Horse	Peppy
387	Wolf	Cranky
388	Koala	Snooty
389	Deer	Smug
390	Octopus	Lazy

391 rows × 2 columns

Now for the names of our types.

In [7]:

left = np.unique(pd.DataFrame(species_personality)[0]).tolist()
pd.DataFrame(left)

Out[7]:

	0
0	Alligator
1	Anteater
2	Bear
3	Bird
4	Bull
5	Cat
6	Chicken
7	Cow
8	Cub
9	Deer
10	Dog
11	Duck
12	Eagle
13	Elephant
14	Frog
15	Goat
16	Gorilla
17	Hamster
18	Hippo
19	Horse
20	Kangaroo
21	Koala
22	Lion
23	Monkey
24	Mouse
25	Octopus
26	Ostrich
27	Penguin
28	Pig
29	Rabbit
30	Rhino
31	Sheep
32	Squirrel
33	Tiger
34	Wolf

In [8]:

right = np.unique(pd.DataFrame(species_personality)[1]).tolist()
pd.DataFrame(right)

Out[8]:

	0
0	Cranky
1	Jock
2	Lazy
3	Normal
4	Peppy
5	Smug
6	Snooty
7	Uchi

Which we can now use to create the matrix.

In [9]:

features= left+right
d = pd.DataFrame(0, index=features, columns=features)

In [ ]:

Our chord diagram will need two inputs: the co-occurrence matrix, and a list of names to label the segments.

We can build a co-occurrence matrix with the following approach. We'll start by creating a list with every type pairing in its original and reversed form.

In [10]:

species_personality = list(itertools.chain.from_iterable((i, i[::-1]) for i in species_personality.values))

In [11]:

for x in species_personality:
    d.at[x[0], x[1]] += 1

In [12]:

Out[12]:

	Alligator	Anteater	Bear	Bird	Bull	Cat	Chicken	Cow	Cub	Deer	...	Tiger	Wolf	Cranky	Jock	Lazy	Normal	Peppy	Smug	Snooty	Uchi
Alligator	0	0	0	0	0	0	0	0	0	0	...	0	0	1	2	2	1	0	0	1	0
Anteater	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	0	1	2	1	1	0
Bear	0	0	0	0	0	0	0	0	0	0	...	0	0	5	1	1	1	2	2	0	3
Bird	0	0	0	0	0	0	0	0	0	0	...	0	0	1	4	2	1	2	2	1	0
Bull	0	0	0	0	0	0	0	0	0	0	...	0	0	3	1	2	0	0	0	0	0
Cat	0	0	0	0	0	0	0	0	0	0	...	0	0	2	3	3	3	5	1	5	1
Chicken	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	2	1	0	1	2	1
Cow	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	1	1	0	2	0
Cub	0	0	0	0	0	0	0	0	0	0	...	0	0	2	2	4	4	2	0	1	1
Deer	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	2	1	0	2	1	2
Dog	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	6	3	2	1	1	1
Duck	0	0	0	0	0	0	0	0	0	0	...	0	0	0	2	4	2	4	1	4	0
Eagle	0	0	0	0	0	0	0	0	0	0	...	0	0	4	2	0	1	0	1	1	0
Elephant	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	4	3	0	0	2	0
Frog	0	0	0	0	0	0	0	0	0	0	...	0	0	3	5	3	2	1	2	1	1
Goat	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	1	2	0	1	1	1
Gorilla	0	0	0	0	0	0	0	0	0	0	...	0	0	3	2	1	0	0	1	1	1
Hamster	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	1	1	1	2	1	0
Hippo	0	0	0	0	0	0	0	0	0	0	...	0	0	2	1	0	1	1	1	1	0
Horse	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	3	2	2	3	2	1
Kangaroo	0	0	0	0	0	0	0	0	0	0	...	0	0	2	0	0	3	0	0	2	1
Koala	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	1	3	0	1	1	1
Lion	0	0	0	0	0	0	0	0	0	0	...	0	0	1	3	1	0	0	2	0	0
Monkey	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	2	1	1	0	1	1
Mouse	0	0	0	0	0	0	0	0	0	0	...	0	0	2	3	1	2	4	1	2	0
Octopus	0	0	0	0	0	0	0	0	0	0	...	0	0	1	0	1	1	0	0	0	0
Ostrich	0	0	0	0	0	0	0	0	0	0	...	0	0	0	1	1	2	1	1	3	1
Penguin	0	0	0	0	0	0	0	0	0	0	...	0	0	1	2	4	1	1	1	2	1
Pig	0	0	0	0	0	0	0	0	0	0	...	0	0	2	3	2	3	2	1	1	1
Rabbit	0	0	0	0	0	0	0	0	0	0	...	0	0	1	2	4	1	8	1	2	1
Rhino	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	1	2	0	0	0	1
Sheep	0	0	0	0	0	0	0	0	0	0	...	0	0	0	1	0	3	1	2	4	2
Squirrel	0	0	0	0	0	0	0	0	0	0	...	0	0	2	1	1	5	3	1	4	1
Tiger	0	0	0	0	0	0	0	0	0	0	...	0	0	1	3	0	0	2	0	1	0
Wolf	0	0	0	0	0	0	0	0	0	0	...	0	0	5	0	0	1	1	1	3	0
Cranky	1	1	5	1	3	2	1	0	2	1	...	1	5	0	0	0	0	0	0	0	0
Jock	2	1	1	4	1	3	1	0	2	1	...	3	0	0	0	0	0	0	0	0	0
Lazy	2	0	1	2	2	3	2	0	4	2	...	0	0	0	0	0	0	0	0	0	0
Normal	1	1	1	1	0	3	1	1	4	1	...	0	1	0	0	0	0	0	0	0	0
Peppy	0	2	2	2	0	5	0	1	2	0	...	2	1	0	0	0	0	0	0	0	0
Smug	0	1	2	2	0	1	1	0	0	2	...	0	1	0	0	0	0	0	0	0	0
Snooty	1	1	0	1	0	5	2	2	1	1	...	1	3	0	0	0	0	0	0	0	0
Uchi	0	0	3	0	0	1	1	0	1	2	...	0	0	0	0	0	0	0	0	0	0

43 rows × 43 columns

Chord Diagram¶

Time to visualise the co-occurrence of types using a chord diagram. We are going to use a list of custom colours that represent the types.

In [13]:

colors =["#ff2200","#ffcc00","#ace6cb","#0057d9","#633366","#73341d","#665f00","#00ffcc","#001433","#e6acda","#ffa280","#eeff00","#336663","#001f73","#ff00aa","#ffd9bf","#f2ffbf","#36ced9","#737399","#73003d","#ff8800","#44ff00","#00a2f2","#6600ff","#ff0044","#99754d","#416633","#004d73","#5e008c","#bf606c","#332200","#60bf60","#acd2e6","#e680ff","#66333a","#3d005c","#6e0060","#99005d","#bd0055","#db2f48","#f05738","#fc7e23","#ffa600"]

In [14]:

names = left + right

Finally, we can put it all together.

In [15]:

Chord(d.values.tolist(), names,colors=colors, wrap_labels=False, margin=40, font_size_large=10).show()

Chord Diagram

In [32]:

Chord(d.values.tolist(), names,credit=True, colors=colors, wrap_labels=False,
      margin=40, font_size_large=7,noun="villagers",
        details_separator="", divide=True, divide_idx=len(left),divide_size=.2, width=850).show()

Chord Diagram

Chord Diagram with Names¶

It would be nice to show a list of Pokémon names when hovering over co-occurring Pokémon types. To do this, we can make use of the optional details parameter.

In [ ]:

Next, we'll create an empty multi-dimensional array with the same shape as our matrix.

In [17]:

details = np.empty((len(names),len(names)),dtype=object)
details_thumbs = np.empty((len(names),len(names)),dtype=object)

Now we can populate the details array with lists of Pokémon names in the correct positions.

In [18]:

for count_x, item_x in enumerate(names):
    for count_y, item_y in enumerate(names):
        details_urls = data[
            (data['species'].isin([item_x, item_y])) &
            (data['personality'].isin([item_y, item_x]))]['url'].to_list()
        
        details_names = data[
            (data['species'].isin([item_x, item_y])) &
            (data['personality'].isin([item_y, item_x]))]['name'].to_list()
        
        urls_names = np.column_stack((details_urls, details_names))
        if(urls_names.size > 0):
            details[count_x][count_y] = details_names
            details_thumbs[count_x][count_y] = details_urls

        else:
            details[count_x][count_y] = []
            details_thumbs[count_x][count_y] = []

details=pd.DataFrame(details).values.tolist()
details_thumbs=pd.DataFrame(details_thumbs).values.tolist()

In [19]:

len(right)

Out[19]:

Finally, we can put it all together but this time with the details matrix passed in.

In [33]:

Chord(d.values.tolist(), names,credit=True, colors=colors, wrap_labels=False,
      padding=0, font_size_large=10,details=details,details_thumbs=details_thumbs,noun="villagers",
        details_separator="", divide=True, divide_idx=len(left),divide_size=.2, width=800).show()

Chord Diagram

In [21]:

np.empty(shape=(6,1)).tolist()

Out[21]:

[[2.291755454583e-312],
 [2.22809558106e-312],
 [2.143215749443e-312],
 [2.37663528627e-312],
 [2.29175545472e-312],
 [0.0]]

In [22]:

Out[22]:

	Alligator	Anteater	Bear	Bird	Bull	Cat	Chicken	Cow	Cub	Deer	...	Tiger	Wolf	Cranky	Jock	Lazy	Normal	Peppy	Smug	Snooty	Uchi
Alligator	0	0	0	0	0	0	0	0	0	0	...	0	0	1	2	2	1	0	0	1	0
Anteater	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	0	1	2	1	1	0
Bear	0	0	0	0	0	0	0	0	0	0	...	0	0	5	1	1	1	2	2	0	3
Bird	0	0	0	0	0	0	0	0	0	0	...	0	0	1	4	2	1	2	2	1	0
Bull	0	0	0	0	0	0	0	0	0	0	...	0	0	3	1	2	0	0	0	0	0
Cat	0	0	0	0	0	0	0	0	0	0	...	0	0	2	3	3	3	5	1	5	1
Chicken	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	2	1	0	1	2	1
Cow	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	1	1	0	2	0
Cub	0	0	0	0	0	0	0	0	0	0	...	0	0	2	2	4	4	2	0	1	1
Deer	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	2	1	0	2	1	2
Dog	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	6	3	2	1	1	1
Duck	0	0	0	0	0	0	0	0	0	0	...	0	0	0	2	4	2	4	1	4	0
Eagle	0	0	0	0	0	0	0	0	0	0	...	0	0	4	2	0	1	0	1	1	0
Elephant	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	4	3	0	0	2	0
Frog	0	0	0	0	0	0	0	0	0	0	...	0	0	3	5	3	2	1	2	1	1
Goat	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	1	2	0	1	1	1
Gorilla	0	0	0	0	0	0	0	0	0	0	...	0	0	3	2	1	0	0	1	1	1
Hamster	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	1	1	1	2	1	0
Hippo	0	0	0	0	0	0	0	0	0	0	...	0	0	2	1	0	1	1	1	1	0
Horse	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	3	2	2	3	2	1
Kangaroo	0	0	0	0	0	0	0	0	0	0	...	0	0	2	0	0	3	0	0	2	1
Koala	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	1	3	0	1	1	1
Lion	0	0	0	0	0	0	0	0	0	0	...	0	0	1	3	1	0	0	2	0	0
Monkey	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	2	1	1	0	1	1
Mouse	0	0	0	0	0	0	0	0	0	0	...	0	0	2	3	1	2	4	1	2	0
Octopus	0	0	0	0	0	0	0	0	0	0	...	0	0	1	0	1	1	0	0	0	0
Ostrich	0	0	0	0	0	0	0	0	0	0	...	0	0	0	1	1	2	1	1	3	1
Penguin	0	0	0	0	0	0	0	0	0	0	...	0	0	1	2	4	1	1	1	2	1
Pig	0	0	0	0	0	0	0	0	0	0	...	0	0	2	3	2	3	2	1	1	1
Rabbit	0	0	0	0	0	0	0	0	0	0	...	0	0	1	2	4	1	8	1	2	1
Rhino	0	0	0	0	0	0	0	0	0	0	...	0	0	1	1	1	2	0	0	0	1
Sheep	0	0	0	0	0	0	0	0	0	0	...	0	0	0	1	0	3	1	2	4	2
Squirrel	0	0	0	0	0	0	0	0	0	0	...	0	0	2	1	1	5	3	1	4	1
Tiger	0	0	0	0	0	0	0	0	0	0	...	0	0	1	3	0	0	2	0	1	0
Wolf	0	0	0	0	0	0	0	0	0	0	...	0	0	5	0	0	1	1	1	3	0
Cranky	1	1	5	1	3	2	1	0	2	1	...	1	5	0	0	0	0	0	0	0	0
Jock	2	1	1	4	1	3	1	0	2	1	...	3	0	0	0	0	0	0	0	0	0
Lazy	2	0	1	2	2	3	2	0	4	2	...	0	0	0	0	0	0	0	0	0	0
Normal	1	1	1	1	0	3	1	1	4	1	...	0	1	0	0	0	0	0	0	0	0
Peppy	0	2	2	2	0	5	0	1	2	0	...	2	1	0	0	0	0	0	0	0	0
Smug	0	1	2	2	0	1	1	0	0	2	...	0	1	0	0	0	0	0	0	0	0
Snooty	1	1	0	1	0	5	2	2	1	1	...	1	3	0	0	0	0	0	0	0	0
Uchi	0	0	3	0	0	1	1	0	1	2	...	0	0	0	0	0	0	0	0	0	0

43 rows × 43 columns

Conclusion¶

Co-occurrence of Anime Genres with Chord Diagrams

Dr. Shahin Rostami

2020-04-21

Preamble¶

In [1]:

import numpy as np                   # for multi-dimensional containers 
import pandas as pd                  # for DataFrames
import itertools
from ast import literal_eval
from chord import Chord

Introduction¶

In this section, we're going to use the MyAnimeList dataset to visualise the co-occurrence of anime genres.

The Dataset¶

The dataset documentation states that we can expect 31 variables per each of the 14478 entries. Let's download the mirrored dataset and have a look for ourselves.

In [2]:

data_url = 'https://datacrayon.com/datasets/anime_list.csv'
data = pd.read_csv(data_url)
data.head()

Out[2]:

	anime_id	title	title_english	title_japanese	title_synonyms	image_url	type	source	episodes	status	...	background	premiered	broadcast	related	producer	licensor	studio	genre	opening_theme	ending_theme
0	11013	Inu x Boku SS	Inu X Boku Secret Service	妖狐×僕SS	Youko x Boku SS	https://myanimelist.cdn-dena.com/images/anime/...	TV	Manga	12	Finished Airing	...	Inu x Boku SS was licensed by Sentai Filmworks...	Winter 2012	Fridays at Unknown	{'Adaptation': [{'mal_id': 17207, 'type': 'man...	Aniplex, Square Enix, Mainichi Broadcasting Sy...	Sentai Filmworks	David Production	Comedy, Supernatural, Romance, Shounen	['"Nirvana" by MUCC']	['#1: "Nirvana" by MUCC (eps 1, 11-12)', '#2: ...
1	2104	Seto no Hanayome	My Bride is a Mermaid	瀬戸の花嫁	The Inland Sea Bride	https://myanimelist.cdn-dena.com/images/anime/...	TV	Manga	26	Finished Airing	...	NaN	Spring 2007	Unknown	{'Adaptation': [{'mal_id': 759, 'type': 'manga...	TV Tokyo, AIC, Square Enix, Sotsu	Funimation	Gonzo	Comedy, Parody, Romance, School, Shounen	['"Romantic summer" by SUN&LUNAR']	['#1: "Ashita e no Hikari (明日への光)" by Asuka Hi...
2	5262	Shugo Chara!! Doki	Shugo Chara!! Doki	しゅごキャラ！！どきっ	Shugo Chara Ninenme, Shugo Chara! Second Year	https://myanimelist.cdn-dena.com/images/anime/...	TV	Manga	51	Finished Airing	...	NaN	Fall 2008	Unknown	{'Adaptation': [{'mal_id': 101, 'type': 'manga...	TV Tokyo, Sotsu	NaN	Satelight	Comedy, Magic, School, Shoujo	['#1: "Minna no Tamago (みんなのたまご)" by Shugo Cha...	['#1: "Rottara Rottara (ロッタラロッタラ)" by Buono! ...
3	721	Princess Tutu	Princess Tutu	プリンセスチュチュ	NaN	https://myanimelist.cdn-dena.com/images/anime/...	TV	Original	38	Finished Airing	...	Princess Tutu aired in two parts. The first pa...	Summer 2002	Fridays at Unknown	{'Adaptation': [{'mal_id': 1581, 'type': 'mang...	Memory-Tech, GANSIS, Marvelous AQL	ADV Films	Hal Film Maker	Comedy, Drama, Magic, Romance, Fantasy	['"Morning Grace" by Ritsuko Okazaki']	['"Watashi No Ai Wa Chiisaikeredo" by Ritsuko ...
4	12365	Bakuman. 3rd Season	Bakuman.	バクマン。	Bakuman Season 3	https://myanimelist.cdn-dena.com/images/anime/...	TV	Manga	25	Finished Airing	...	NaN	Fall 2012	Unknown	{'Adaptation': [{'mal_id': 9711, 'type': 'mang...	NHK, Shueisha	NaN	J.C.Staff	Comedy, Drama, Romance, Shounen	['#1: "Moshimo no Hanashi (もしもの話)" by nano.RIP...	['#1: "Pride on Everyday" by Sphere (eps 1-13)...

5 rows × 31 columns

It looks good so far, but let's confirm the 31 variables against 14478 samples from the documentation.

In [3]:

data.shape

Out[3]:

(14478, 31)

Perfect, that's exactly what we were expecting.

Data Wrangling¶

We need to do a bit of data wrangling before we can visualise our data. We can see from the column names there's a single column for genres, containing comma separated values.

Let's convert them to lists of strings.

In [4]:

def get_list(x):
    if isinstance(x, int):
        return []
    if isinstance(x,str):
        result = [s.strip() for s in x.split(',')]
        return sorted(result)

    return []

In [5]:

genres = data['genre'].apply(get_list)
pd.DataFrame(genres)

Out[5]:

	genre
0	[Comedy, Romance, Shounen, Supernatural]
1	[Comedy, Parody, Romance, School, Shounen]
2	[Comedy, Magic, School, Shoujo]
3	[Comedy, Drama, Fantasy, Magic, Romance]
4	[Comedy, Drama, Romance, Shounen]
...	...
14473	[Kids]
14474	[Comedy]
14475	[Action, Adventure, Fantasy, Sci-Fi]
14476	[Fantasy, Kids]
14477	[Comedy]

14478 rows × 1 columns

Without further investigation, we can see that we have at least a few empty list values, [], and a few single-entry lists in the table above, so let's remove all samples which contain an empty or single-entry list.

In [6]:

genres = genres[genres.str.len() > 1]
pd.DataFrame(genres)

Out[6]:

	genre
0	[Comedy, Romance, Shounen, Supernatural]
1	[Comedy, Parody, Romance, School, Shounen]
2	[Comedy, Magic, School, Shoujo]
3	[Comedy, Drama, Fantasy, Magic, Romance]
4	[Comedy, Drama, Romance, Shounen]
...	...
14467	[Drama, Kids]
14469	[Kids, School]
14471	[Drama, Fantasy, Kids]
14475	[Action, Adventure, Fantasy, Sci-Fi]
14476	[Fantasy, Kids]

10974 rows × 1 columns

Our chord diagram will need two inputs: the co-occurrence matrix, and a list of names to label the segments.

We can build a co-occurrence matrix with the following approach. We'll start by cgetting all combinations within each list.

In [7]:

genres = [list(itertools.combinations(i,2)) for i in genres]
pd.DataFrame(genres)

Out[7]:

	0	1	2	3	4	5	6	7	8	9	...	68	69	70	71	72	73	74	75	76	77
0	(Comedy, Romance)	(Comedy, Shounen)	(Comedy, Supernatural)	(Romance, Shounen)	(Romance, Supernatural)	(Shounen, Supernatural)	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
1	(Comedy, Parody)	(Comedy, Romance)	(Comedy, School)	(Comedy, Shounen)	(Parody, Romance)	(Parody, School)	(Parody, Shounen)	(Romance, School)	(Romance, Shounen)	(School, Shounen)	...	None	None	None	None	None	None	None	None	None	None
2	(Comedy, Magic)	(Comedy, School)	(Comedy, Shoujo)	(Magic, School)	(Magic, Shoujo)	(School, Shoujo)	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
3	(Comedy, Drama)	(Comedy, Fantasy)	(Comedy, Magic)	(Comedy, Romance)	(Drama, Fantasy)	(Drama, Magic)	(Drama, Romance)	(Fantasy, Magic)	(Fantasy, Romance)	(Magic, Romance)	...	None	None	None	None	None	None	None	None	None	None
4	(Comedy, Drama)	(Comedy, Romance)	(Comedy, Shounen)	(Drama, Romance)	(Drama, Shounen)	(Romance, Shounen)	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
10969	(Drama, Kids)	None	None	None	None	None	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
10970	(Kids, School)	None	None	None	None	None	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
10971	(Drama, Fantasy)	(Drama, Kids)	(Fantasy, Kids)	None	None	None	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
10972	(Action, Adventure)	(Action, Fantasy)	(Action, Sci-Fi)	(Adventure, Fantasy)	(Adventure, Sci-Fi)	(Fantasy, Sci-Fi)	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
10973	(Fantasy, Kids)	None	None	None	None	None	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None

10974 rows × 78 columns

Now we will flatten the nested lists, this will give us all the genre pairings in original and reversed order.

In [8]:

genres = list(itertools.chain.from_iterable((i, i[::-1]) for c_ in genres for i in c_))
pd.DataFrame(genres)

Out[8]:

	0	1
0	Comedy	Romance
1	Romance	Comedy
2	Comedy	Shounen
3	Shounen	Comedy
4	Comedy	Supernatural
...	...	...
119691	Sci-Fi	Adventure
119692	Fantasy	Sci-Fi
119693	Sci-Fi	Fantasy
119694	Fantasy	Kids
119695	Kids	Fantasy

119696 rows × 2 columns

Which we can now use to create the matrix.

In [9]:

matrix = pd.pivot_table(
    pd.DataFrame(genres), index=0, columns=1, aggfunc="size", fill_value=0
).values.tolist()

We can list this using a DataFrame for better presentation.

In [10]:

pd.DataFrame(matrix)

Out[10]:

	0	1	2	3	4	5	6	7	8	9	...	33	34	35	36	37	38	39	40	41	42
0	0	1022	34	985	12	165	550	201	860	103	...	13	38	232	141	390	475	32	56	3	2
1	1022	0	22	994	6	97	428	67	1054	80	...	7	70	134	39	136	229	6	10	0	0
2	34	22	0	11	0	0	12	0	1	8	...	0	2	1	36	0	1	0	0	0	0
3	985	994	11	0	28	103	499	500	952	65	...	41	835	79	246	249	439	11	52	12	11
4	12	6	0	28	0	1	21	3	18	1	...	1	1	2	1	0	16	4	0	0	0
5	165	97	0	103	1	0	41	24	166	8	...	0	3	1	1	25	194	4	11	0	1
6	550	428	12	499	21	41	0	54	369	19	...	37	336	137	148	58	236	36	23	23	1
7	201	67	0	500	3	24	54	0	136	11	...	0	35	10	22	45	97	0	13	0	8
8	860	1054	1	952	18	166	369	136	0	101	...	11	132	17	11	110	360	7	33	0	3
9	103	80	8	65	1	8	19	11	101	0	...	0	20	3	7	1	14	6	0	0	0
10	73	19	0	225	0	18	53	156	74	1	...	1	20	6	1	10	55	0	11	0	2
11	41	15	0	52	2	57	28	0	68	0	...	0	1	4	3	5	63	0	3	11	32
12	224	214	1	195	3	56	335	7	184	4	...	7	82	3	6	15	135	3	3	2	0
13	139	62	1	71	30	74	88	14	94	2	...	3	1	11	2	10	201	21	35	1	0
14	17	8	0	39	0	4	33	0	18	3	...	3	29	0	3	0	17	0	3	0	0
15	134	476	23	596	1	22	224	3	588	34	...	0	122	19	35	36	48	0	5	0	0
16	316	278	0	407	0	57	131	89	538	19	...	4	60	5	0	52	168	7	7	0	0
17	236	103	0	101	1	20	41	26	72	3	...	1	15	2	20	86	26	0	0	0	1
18	623	302	10	224	8	4	188	35	73	16	...	0	13	182	13	44	22	1	0	1	0
19	304	90	0	61	3	14	193	19	52	5	...	0	9	137	4	18	31	2	6	1	1
20	64	35	3	137	51	4	114	9	89	7	...	3	113	28	19	6	23	0	5	3	1
21	206	154	0	210	11	19	140	10	79	14	...	5	24	6	3	44	188	48	23	0	0
22	88	27	2	453	8	6	6	40	55	15	...	3	29	11	11	44	18	1	3	1	2
23	105	73	5	114	2	1	29	6	3	0	...	1	5	3	5	1	11	9	1	0	0
24	64	24	0	32	29	6	117	6	32	16	...	4	12	7	2	2	74	38	0	1	0
25	277	244	2	838	6	64	606	243	306	13	...	45	241	45	39	32	224	9	24	21	3
26	114	38	0	53	0	10	39	8	23	0	...	0	1	0	0	11	21	0	1	2	0
27	227	43	1	831	3	25	238	209	125	29	...	13	361	7	135	62	140	4	16	3	6
28	1143	695	10	676	20	32	464	110	262	40	...	6	72	377	41	143	98	18	6	3	2
29	256	105	21	373	2	20	144	90	65	12	...	0	153	24	40	26	102	16	13	0	3
30	76	77	0	242	1	31	191	0	205	1	...	15	119	1	23	9	66	1	14	0	0
31	13	1	0	33	1	0	20	14	9	0	...	0	22	0	0	2	4	0	0	0	2
32	809	675	19	963	1	73	266	114	431	63	...	0	85	63	276	183	249	7	24	0	0
33	13	7	0	41	1	0	37	0	11	0	...	0	10	0	2	1	16	1	4	1	0
34	38	70	2	835	1	3	336	35	132	20	...	10	0	7	39	8	93	1	1	2	0
35	232	134	1	79	2	1	137	10	17	3	...	0	7	0	3	9	7	0	0	0	0
36	141	39	36	246	1	1	148	22	11	7	...	2	39	3	0	11	3	0	0	2	0
37	390	136	0	249	0	25	58	45	110	1	...	1	8	9	11	0	90	3	8	0	0
38	475	229	1	439	16	194	236	97	360	14	...	16	93	7	3	90	0	39	86	2	0
39	32	6	0	11	4	4	36	0	7	6	...	1	1	0	0	3	39	0	1	0	0
40	56	10	0	52	0	11	23	13	33	0	...	4	1	0	0	8	86	1	0	0	0
41	3	0	0	12	0	0	23	0	0	0	...	1	2	0	2	0	2	0	0	0	0
42	2	0	0	11	0	1	1	8	3	0	...	0	0	0	0	0	0	0	0	0	0

43 rows × 43 columns

Now for the names of our genres.

In [11]:

names = np.unique(genres).tolist()
pd.DataFrame(names)

Out[11]:

	0
0	Action
1	Adventure
2	Cars
3	Comedy
4	Dementia
5	Demons
6	Drama
7	Ecchi
8	Fantasy
9	Game
10	Harem
11	Hentai
12	Historical
13	Horror
14	Josei
15	Kids
16	Magic
17	Martial Arts
18	Mecha
19	Military
20	Music
21	Mystery
22	Parody
23	Police
24	Psychological
25	Romance
26	Samurai
27	School
28	Sci-Fi
29	Seinen
30	Shoujo
31	Shoujo Ai
32	Shounen
33	Shounen Ai
34	Slice of Life
35	Space
36	Sports
37	Super Power
38	Supernatural
39	Thriller
40	Vampire
41	Yaoi
42	Yuri

We may wish to remove some genres for our visualisation. The example below will remove a single genre from the co-occurrence matrix and list of names, however, if you add more genre names to the discarded_categories list it will work for them too.

In [12]:

matrix = pd.DataFrame(matrix)
names = pd.DataFrame(names)

discarded_categories = ["Hentai", "Yaoi", "Yuri", "Ecchi",
                        "Shounen Ai", "Shoujo Ai"]

discard_mask = names.isin(discarded_categories).values
discard_indices = names[discard_mask].index

for drop_idx in discard_indices:
    matrix = matrix.drop(drop_idx, axis=1)
    matrix = matrix.drop(drop_idx, axis=0)
    names = names.drop(drop_idx, axis=0)

Chord Diagram¶

Time to visualise the co-occurrence of genres using a chord diagram. We are going to use a list of custom colours that represent the genres.

In [13]:

colors = ["#660000", "#734139", "#e59173", "#ff4400", "#332b26", "#593000",
          "#998773", "#d97400", "#8c5e00", "#f2ca79", "#ffcc00", "#59562d",
          "#736b00", "#c2cc33", "#245900", "#8cff40", "#269926", "#ace6ac",
          "#40ffa6", "#336655", "#008c5e", "#39e6da", "#ace6e2", "#566d73",
          "#39c3e6", "#1d5673", "#3d9df2", "#163159", "#acc3e6", "#000f73",
          "#565a73", "#000033", "#8273e6", "#6d00cc", "#633366", "#e2ace6",
          "#f23de6", "#cc0088", "#590024", "#cc0036", "#f27999", "#e6acb4"];

Finally, we can put it all together.

In [30]:

Chord(
    matrix.values.tolist(),
    names.values.tolist(),
    padding=0.03,
    colors=colors,
    credit=True,
    noun = "Anime",
    font_size_large = "16px"
).show()

Chord Diagram

Conclusion¶

Co-occurrence of Movie Genres with Chord Diagrams

Dr. Shahin Rostami

2020-04-16

Preamble¶

In [1]:

import numpy as np                   # for multi-dimensional containers 
import pandas as pd                  # for DataFrames
import itertools
from ast import literal_eval
from chord import Chord

Introduction¶

In this section, we're going to use the TMDB 5000 Movie Dataset dataset to visualise the co-occurrence of movie genres.

The Dataset¶

The dataset documentation states that we can expect 20 variables per each of the 4803 movies. Let's download the mirrored dataset and have a look for ourselves.

In [2]:

data_url = 'https://datacrayon.com/datasets/tmdb_5000_movies.csv'
data = pd.read_csv(data_url)
data.head()

Out[2]:

	budget	genres	homepage	id	keywords	original_language	original_title	overview	popularity	production_companies	production_countries	release_date	revenue	runtime	spoken_languages	status	tagline	title	vote_average	vote_count
0	237000000	[{"id": 28, "name": "Action"}, {"id": 12, "nam...	http://www.avatarmovie.com/	19995	[{"id": 1463, "name": "culture clash"}, {"id":...	en	Avatar	In the 22nd century, a paraplegic Marine is di...	150.437577	[{"name": "Ingenious Film Partners", "id": 289...	[{"iso_3166_1": "US", "name": "United States o...	2009-12-10	2787965087	162.0	[{"iso_639_1": "en", "name": "English"}, {"iso...	Released	Enter the World of Pandora.	Avatar	7.2	11800
1	300000000	[{"id": 12, "name": "Adventure"}, {"id": 14, "...	http://disney.go.com/disneypictures/pirates/	285	[{"id": 270, "name": "ocean"}, {"id": 726, "na...	en	Pirates of the Caribbean: At World's End	Captain Barbossa, long believed to be dead, ha...	139.082615	[{"name": "Walt Disney Pictures", "id": 2}, {"...	[{"iso_3166_1": "US", "name": "United States o...	2007-05-19	961000000	169.0	[{"iso_639_1": "en", "name": "English"}]	Released	At the end of the world, the adventure begins.	Pirates of the Caribbean: At World's End	6.9	4500
2	245000000	[{"id": 28, "name": "Action"}, {"id": 12, "nam...	http://www.sonypictures.com/movies/spectre/	206647	[{"id": 470, "name": "spy"}, {"id": 818, "name...	en	Spectre	A cryptic message from Bond’s past sends him o...	107.376788	[{"name": "Columbia Pictures", "id": 5}, {"nam...	[{"iso_3166_1": "GB", "name": "United Kingdom"...	2015-10-26	880674609	148.0	[{"iso_639_1": "fr", "name": "Fran\u00e7ais"},...	Released	A Plan No One Escapes	Spectre	6.3	4466
3	250000000	[{"id": 28, "name": "Action"}, {"id": 80, "nam...	http://www.thedarkknightrises.com/	49026	[{"id": 849, "name": "dc comics"}, {"id": 853,...	en	The Dark Knight Rises	Following the death of District Attorney Harve...	112.312950	[{"name": "Legendary Pictures", "id": 923}, {"...	[{"iso_3166_1": "US", "name": "United States o...	2012-07-16	1084939099	165.0	[{"iso_639_1": "en", "name": "English"}]	Released	The Legend Ends	The Dark Knight Rises	7.6	9106
4	260000000	[{"id": 28, "name": "Action"}, {"id": 12, "nam...	http://movies.disney.com/john-carter	49529	[{"id": 818, "name": "based on novel"}, {"id":...	en	John Carter	John Carter is a war-weary, former military ca...	43.926995	[{"name": "Walt Disney Pictures", "id": 2}]	[{"iso_3166_1": "US", "name": "United States o...	2012-03-07	284139100	132.0	[{"iso_639_1": "en", "name": "English"}]	Released	Lost in our world, found in another.	John Carter	6.1	2124

It looks good so far, but let's confirm the 20 variables against 4803 samples from the documentation.

In [3]:

data.shape

Out[3]:

(4803, 20)

Perfect, that's exactly what we were expecting.

Data Wrangling¶

We need to do a bit of data wrangling before we can visualise our data. We can see from the column names there's a single column for genres, containing a string representation of a dictionary.

The first thing we need to do is evaluate these from strings into a type we can work with.

In [4]:

genres = data['genres'].apply(literal_eval)
genres

Out[4]:

0       [{'id': 28, 'name': 'Action'}, {'id': 12, 'nam...
1       [{'id': 12, 'name': 'Adventure'}, {'id': 14, '...
2       [{'id': 28, 'name': 'Action'}, {'id': 12, 'nam...
3       [{'id': 28, 'name': 'Action'}, {'id': 80, 'nam...
4       [{'id': 28, 'name': 'Action'}, {'id': 12, 'nam...
                              ...                        
4798    [{'id': 28, 'name': 'Action'}, {'id': 80, 'nam...
4799    [{'id': 35, 'name': 'Comedy'}, {'id': 10749, '...
4800    [{'id': 35, 'name': 'Comedy'}, {'id': 18, 'nam...
4801                                                   []
4802                  [{'id': 99, 'name': 'Documentary'}]
Name: genres, Length: 4803, dtype: object

The genres are now in lists of dictionaries. Let's convert them to lists of strings.

In [5]:

def get_list(x):
    if isinstance(x, list):
        names = [i['name'] for i in x]
        return sorted(names)

    return []

In [6]:

genres = genres.apply(get_list)
pd.DataFrame(genres)

Out[6]:

	genres
0	[Action, Adventure, Fantasy, Science Fiction]
1	[Action, Adventure, Fantasy]
2	[Action, Adventure, Crime]
3	[Action, Crime, Drama, Thriller]
4	[Action, Adventure, Science Fiction]
...	...
4798	[Action, Crime, Thriller]
4799	[Comedy, Romance]
4800	[Comedy, Drama, Romance, TV Movie]
4801	[]
4802	[Documentary]

4803 rows × 1 columns

Without further investigation, we can see that we have at least a few empty list values, [], in the table above, so we can remove all samples which contain an empty list.

In [7]:

genres = genres[genres.str.len() > 0]
pd.DataFrame(genres)

Out[7]:

	genres
0	[Action, Adventure, Fantasy, Science Fiction]
1	[Action, Adventure, Fantasy]
2	[Action, Adventure, Crime]
3	[Action, Crime, Drama, Thriller]
4	[Action, Adventure, Science Fiction]
...	...
4797	[Foreign, Thriller]
4798	[Action, Crime, Thriller]
4799	[Comedy, Romance]
4800	[Comedy, Drama, Romance, TV Movie]
4802	[Documentary]

4775 rows × 1 columns

Our chord diagram will need two inputs: the co-occurrence matrix, and a list of names to label the segments.

We can build a co-occurrence matrix with the following approach. We'll start by getting all combinations within each list.

In [8]:

genres = [list(itertools.combinations(i,2)) for i in genres]
pd.DataFrame(genres)

Out[8]:

	0	1	2	3	4	5	6	7	8	9	...	11	12	13	14	15	16	17	18	19	20
0	(Action, Adventure)	(Action, Fantasy)	(Action, Science Fiction)	(Adventure, Fantasy)	(Adventure, Science Fiction)	(Fantasy, Science Fiction)	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
1	(Action, Adventure)	(Action, Fantasy)	(Adventure, Fantasy)	None	None	None	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
2	(Action, Adventure)	(Action, Crime)	(Adventure, Crime)	None	None	None	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
3	(Action, Crime)	(Action, Drama)	(Action, Thriller)	(Crime, Drama)	(Crime, Thriller)	(Drama, Thriller)	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
4	(Action, Adventure)	(Action, Science Fiction)	(Adventure, Science Fiction)	None	None	None	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
4770	(Foreign, Thriller)	None	None	None	None	None	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
4771	(Action, Crime)	(Action, Thriller)	(Crime, Thriller)	None	None	None	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
4772	(Comedy, Romance)	None	None	None	None	None	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
4773	(Comedy, Drama)	(Comedy, Romance)	(Comedy, TV Movie)	(Drama, Romance)	(Drama, TV Movie)	(Romance, TV Movie)	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None
4774	None	None	None	None	None	None	None	None	None	None	...	None	None	None	None	None	None	None	None	None	None

4775 rows × 21 columns

Now we will flatten the nested lists, this will give us all the genre pairings in original and reversed order.

In [9]:

genres = list(itertools.chain.from_iterable((i, i[::-1]) for c_ in genres for i in c_))
pd.DataFrame(genres)

Out[9]:

	0	1
0	Action	Adventure
1	Adventure	Action
2	Action	Fantasy
3	Fantasy	Action
4	Action	Science Fiction
...	...	...
24655	Romance	Drama
24656	Drama	TV Movie
24657	TV Movie	Drama
24658	Romance	TV Movie
24659	TV Movie	Romance

24660 rows × 2 columns

Which we can now use to create the matrix.

In [10]:

matrix = pd.pivot_table(
    pd.DataFrame(genres), index=0, columns=1, aggfunc="size", fill_value=0
).values.tolist()

We can list this using a DataFrame for better presentation.

In [11]:

pd.DataFrame(matrix)

Out[11]:

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19
0	0	465	26	258	276	3	339	62	144	5	58	76	9	57	63	277	1	547	55	35
1	465	0	114	223	56	2	183	211	190	2	27	20	7	37	66	205	0	203	30	22
2	26	114	0	125	0	0	19	195	61	1	0	0	14	1	8	30	1	3	1	2
3	258	223	125	0	180	11	576	299	166	9	11	78	84	27	484	109	4	113	8	17
4	276	56	0	180	0	1	381	8	10	4	13	36	11	105	47	18	1	414	4	9
5	3	2	0	11	1	0	7	5	0	2	6	1	15	1	0	0	0	1	0	0
6	339	183	19	576	381	7	0	121	99	27	175	84	106	175	603	102	5	554	118	34
7	62	211	195	299	8	5	121	0	149	3	1	1	32	6	52	58	3	7	0	3
8	144	190	61	166	10	0	99	149	0	0	0	53	11	19	64	85	0	63	3	2
9	5	2	1	9	4	2	27	3	0	0	3	0	0	0	9	0	0	3	1	0
10	58	27	0	11	13	6	175	1	0	3	0	1	6	3	30	0	0	21	59	8
11	76	20	0	78	36	1	84	1	53	0	1	0	3	91	15	95	1	291	1	1
12	9	7	14	84	11	15	106	32	11	0	6	3	0	3	61	2	1	5	1	3
13	57	37	1	27	105	1	175	6	19	0	3	91	3	0	24	47	0	242	3	2
14	63	66	8	484	47	0	603	52	64	9	30	15	61	24	0	31	3	64	26	12
15	277	205	30	109	18	0	102	58	85	0	0	95	2	47	31	0	0	211	2	1
16	1	0	1	4	1	0	5	3	0	0	0	1	1	0	3	0	0	1	0	0
17	547	203	3	113	414	1	554	7	63	3	21	291	5	242	64	211	1	0	24	7
18	55	30	1	8	4	0	118	0	3	1	59	1	1	3	26	2	0	24	0	3
19	35	22	2	17	9	0	34	3	2	0	8	1	3	2	12	1	0	7	3	0

Now for the names of our genres.

In [12]:

names = np.unique(genres).tolist()
pd.DataFrame(names)

Out[12]:

	0
0	Action
1	Adventure
2	Animation
3	Comedy
4	Crime
5	Documentary
6	Drama
7	Family
8	Fantasy
9	Foreign
10	History
11	Horror
12	Music
13	Mystery
14	Romance
15	Science Fiction
16	TV Movie
17	Thriller
18	War
19	Western

Chord Diagram¶

Time to visualise the co-occurrence of genres using a chord diagram. We are going to use a list of custom colours that represent the genres.

In [13]:

colors = ["#e6194B", "#3cb44b", "#ffe119", "#4363d8", "#f58231",
    "#911eb4", "#42d4f4", "#f032e6", "#bfef45", "#fabebe",
    "#469990", "#e6beff", "#9A6324", "#fffac8", "#800000",
    "#aaffc3", "#a9a9a9", "#ffd8b1", "#000075", "#a9a9a9",];

Finally, we can put it all together.

In [14]:

Chord(
    matrix,
    names,
    colors=colors,
    wrap_labels=False,
    margin=50
).show()

Chord Diagram

Conclusion¶

Interactive Chord Diagrams

Dr. Shahin Rostami

2020-04-15

Preamble¶

In [2]:

:dep chord = {Version = "0.8.1"}
use chord::{Chord, Plot};

Introduction¶

Chord diagrams are useful when trying to convey relationships between different entities, and they can be beautiful and eye-catching.

Get Chord Pro¶

Click here to get lifetime access to the full-featured chord visualization API, producing beautiful interactive visualizations, e.g. those featured on the front page of Reddit.

Produce beautiful interactive Chord diagrams.
Customize colours and font-sizes.
Access Divided mode, enabling two sides to your diagram.
Symmetric and Asymmetric modes,
Add images and text on hover,
Access finer-customisations including HTML injection.
Allows commercial use without open source requirement.
Currently supports Python, JavaScript, and Rust, with many more to come (accepting requests).

The Chord Package¶

With Python in mind, there are many libraries available for creating Chord diagrams, such as Plotly, Bokeh, and a few that are lesser-known. However, I wanted to use the implementation from d3 because it can be customised to be highly interactive and to look beautiful.

I couldn't find anything that ticked all the boxes, so I made a wrapper around d3-chord myself. It took some time to get it working, but I wanted to hide away everything behind a single constructor and method call. The tricky part was enabling multiple chord diagrams on the same page, and then loading resources in a way that would support Jupyter Notebooks.

You can get the package either from PyPi using pip install chord or from the GitHub repository. With your processed data, you should be able to plot something beautiful with just a single line, Chord(data, names).show(). To enable the pro features of the chord package, get Chord Pro.

The Chord Crate¶

I wasn't able to find any Rust crates for plotting chord diagrams, so I ported my own from Python to Rust.

The Dataset¶

The focus for this section will be the demonstration of the chord crate. To keep it simple, we will use synthetic data that illustrates the co-occurrences between movie genres within the same movie.

In [3]:

let matrix: Vec<Vec<f64>> = vec![
    vec![0., 5., 6., 4., 7., 4.],
    vec![5., 0., 5., 4., 6., 5.],
    vec![6., 5., 0., 4., 5., 5.],
    vec![4., 4., 4., 0., 5., 5.],
    vec![7., 6., 5., 5., 0., 4.],
    vec![4., 5., 5., 5., 4., 0.],
];

let names: Vec<String> = vec![
    "Action",
    "Adventure",
    "Comedy",
    "Drama",
    "Fantasy",
    "Thriller",
]
.into_iter()
.map(String::from)
.collect();

Chord Diagrams¶

Let's see what the Chord defaults produce when we invoke the show() method.

In [6]:

Chord {
    matrix: matrix.clone(),
    names: names.clone(),
    wrap_labels: true,
    ..Chord::default()
}
.show();

Out[6]:

Chord Diagram

Different Colours¶

The defaults are nice, but what if we want different colours? You can pass in almost anything from d3-scale-chromatic, or you could pass in a list of hexadecimal colour codes.

In [5]:

Chord {
    matrix: matrix.clone(),
    names: names.clone(),
    wrap_labels: true,
    colors: vec![String::from("d3.schemeSet2")],
    ..Chord::default()
}
.show();

Out[5]:

Chord Diagram

In [6]:

Chord {
    matrix: matrix.clone(),
    names: names.clone(),
    wrap_labels: true,
    colors: vec![String::from(format!("d3.schemeGnBu[{:?}]",names.len()))],
    ..Chord::default()
}
.show();

Out[6]:

Chord Diagram

In [7]:

Chord {
    matrix: matrix.clone(),
    names: names.clone(),
    wrap_labels: true,
    colors: vec![String::from("d3.schemeSet3")],
    ..Chord::default()
}
.show();

Out[7]:

Chord Diagram

In [8]:

Chord {
    matrix: matrix.clone(),
    names: names.clone(),
    wrap_labels: true,
    colors: vec![String::from(format!("d3.schemePuRd[{:?}]",names.len()))],
    ..Chord::default()
}
.show();

Out[8]:

Chord Diagram

In [9]:

Chord {
    matrix: matrix.clone(),
    names: names.clone(),
    wrap_labels: true,
    colors: vec![String::from(format!("d3.schemeYlGnBu[{:?}]",names.len()))],
    ..Chord::default()
}
.show();

Out[9]:

Chord Diagram

In [10]:

let hex_colours : Vec<String> = vec!["#222222", "#333333", "#4c4c4c", "#666666", "#848484", "#9a9a9a"].into_iter()
.map(String::from)
.collect();

Chord {
    matrix: matrix.clone(),
    names: names.clone(),
    wrap_labels: true,
    colors: hex_colours,
    ..Chord::default()
}
.show();

Out[10]:

Chord Diagram

Label Styling¶

We can disable wrapped labels, and even change the colour.

In [11]:

Chord {
    matrix: matrix.clone(),
    names: names.clone(),
    wrap_labels: false,
    label_color:"#4c40bf".to_string(),
    ..Chord::default()
}
.show();

Out[11]:

Chord Diagram

Opacity¶

We can also change the default opacity of the relationships.

In [12]:

Chord {
    matrix: matrix.clone(),
    names: names.clone(),
    opacity: 0.1,
    ..Chord::default()
}
.show();

Out[12]:

Chord Diagram

Width¶

We can also change the maximum width the plot.

In [14]:

Chord {
    matrix: matrix.clone(),
    names: names.clone(),
    width: 400.0,
    wrap_labels: true,
    ..Chord::default()
}
.show()

Out[14]:

Chord Diagram

Conclusion¶

In this section, we've introduced the chord diagram and chord crate. We used the crate and some synthetic data to demonstrate several chord diagram visualisations with different configurations. The chord Python crate is available for free from crates.io or from the GitHub repository.

Tintarev, N., Rostami, S., & Smyth, B. (2018, April). Knowing the unknown: visualising consumption blind-spots in recommender systems. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing (pp. 1396-1399). ↩

Co-occurrence of Pokemon Types (Gen 1-8) with Chord Diagrams

Dr. Shahin Rostami

2020-04-08

Preamble¶

In [1]:

import numpy as np                   # for multi-dimensional containers 
import pandas as pd                  # for DataFrames
import itertools
from chord import Chord

Introduction¶

In previous sections, we visualised co-occurrences of Pokémon type. Whilst it was interesting to look at, the dataset only contained Pokémon from the first six geerations. In this section, we're going to use the Pokemon with stats Generation 8 dataset to visualise the co-occurrence of Pokémon types from generations one to eight.

The Dataset¶

The dataset documentation states that we can expect 51 variables per each of the 1028 Pokémon of the first eight generations.

Let's download the mirrored dataset and have a look for ourselves.

In [2]:

data_url = 'https://shahinrostami.com/datasets/pokemon_gen_1_to_8.csv'
data = pd.read_csv(data_url)
data.head()

Out[2]:

	Unnamed: 0	pokedex_number	name	german_name	japanese_name	generation	status	species	type_number	type_1	...	against_ground	against_flying	against_psychic	against_bug	against_rock	against_ghost	against_dragon	against_dark	against_steel	against_fairy
0	0	1	Bulbasaur	Bisasam	フシギダネ (Fushigidane)	1	Normal	Seed Pokémon	2	Grass	...	1.0	2.0	2.0	1.0	1.0	1.0	1.0	1.0	1.0	0.5
1	1	2	Ivysaur	Bisaknosp	フシギソウ (Fushigisou)	1	Normal	Seed Pokémon	2	Grass	...	1.0	2.0	2.0	1.0	1.0	1.0	1.0	1.0	1.0	0.5
2	2	3	Venusaur	Bisaflor	フシギバナ (Fushigibana)	1	Normal	Seed Pokémon	2	Grass	...	1.0	2.0	2.0	1.0	1.0	1.0	1.0	1.0	1.0	0.5
3	3	3	Mega Venusaur	Bisaflor	フシギバナ (Fushigibana)	1	Normal	Seed Pokémon	2	Grass	...	1.0	2.0	2.0	1.0	1.0	1.0	1.0	1.0	1.0	0.5
4	4	4	Charmander	Glumanda	ヒトカゲ (Hitokage)	1	Normal	Lizard Pokémon	1	Fire	...	2.0	1.0	1.0	0.5	2.0	1.0	1.0	1.0	0.5	0.5

5 rows × 51 columns

It looks good so far, but let's confirm the 51 variables against 1028 samples from the documentation.

In [3]:

data.shape

Out[3]:

(1028, 51)

Perfect, that's exactly what we were expecting.

Data Wrangling¶

We need to do a bit of data wrangling before we can visualise our data. We can see from the columns names that the Pokémon types are split between the columns type_1 and type_2.

In [4]:

pd.DataFrame(data.columns.values.tolist()).head(20)

Out[4]:

	0
0	Unnamed: 0
1	pokedex_number
2	name
3	german_name
4	japanese_name
5	generation
6	status
7	species
8	type_number
9	type_1
10	type_2
11	height_m
12	weight_kg
13	abilities_number
14	ability_1
15	ability_2
16	ability_hidden
17	total_points
18	hp
19	attack

So let's select just these two columns and work with a list containing only them as we move forward.

In [5]:

types = pd.DataFrame(data[['type_1', 'type_2']].values)
types

Out[5]:

	0	1
0	Grass	Poison
1	Grass	Poison
2	Grass	Poison
3	Grass	Poison
4	Fire	NaN
...	...	...
1023	Fairy	NaN
1024	Fighting	Steel
1025	Fighting	NaN
1026	Poison	Dragon
1027	Poison	Dragon

1028 rows × 2 columns

In [6]:

types = types.dropna()

We can also see an instance where the type Fighting at index $1014$ is followed by \n. We'll strip all these out before continuing.

In [7]:

types = types.replace('\n','', regex=True)
types

Out[7]:

	0	1
0	Grass	Poison
1	Grass	Poison
2	Grass	Poison
3	Grass	Poison
6	Fire	Flying
...	...	...
1021	Dragon	Ghost
1022	Fairy	Steel
1024	Fighting	Steel
1026	Poison	Dragon
1027	Poison	Dragon

542 rows × 2 columns

Our chord diagram will need two inputs: the co-occurrence matrix, and a list of names to label the segments.

We can build a co-occurrence matrix with the following approach. We'll start by creating a list with every type pairing in its original and reversed form.

In [8]:

types = list(itertools.chain.from_iterable((i, i[::-1]) for i in types.values))

Which we can now use to create the matrix.

In [9]:

matrix = pd.pivot_table(
    pd.DataFrame(types), index=0, columns=1, aggfunc="size", fill_value=0
).values.tolist()

We can list this using a DataFrame for better presentation.

In [10]:

pd.DataFrame(matrix)

Out[10]:

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
0	0	0	0	4	2	4	4	14	1	6	2	2	0	13	2	5	7	5
1	0	0	4	2	3	3	4	5	3	3	3	2	5	5	3	2	2	7
2	0	4	0	3	1	2	3	8	5	5	9	3	1	4	5	2	2	3
3	4	2	3	0	2	0	1	6	1	1	1	2	2	3	1	3	4	3
4	2	3	1	2	0	0	0	2	1	5	0	1	5	1	9	3	5	4
5	4	3	2	0	0	0	7	1	1	3	0	1	4	2	6	1	4	3
6	4	4	3	1	0	7	0	7	5	0	4	1	2	2	3	3	1	1
7	14	5	8	6	2	1	7	0	3	7	4	2	27	3	7	6	3	8
8	1	3	5	1	1	1	5	3	0	12	6	1	0	4	3	0	4	2
9	6	3	5	1	5	3	0	7	12	0	1	3	2	15	3	2	3	3
10	2	3	9	1	0	0	4	4	6	1	0	3	1	2	2	9	6	10
11	2	2	3	2	1	1	1	2	1	3	3	0	0	0	4	2	2	7
12	0	5	1	2	5	4	2	27	0	2	1	0	0	0	5	0	0	1
13	13	5	4	3	1	2	2	3	4	15	2	0	0	0	0	1	0	6
14	2	3	5	1	9	6	3	7	3	3	2	4	5	0	0	2	9	6
15	5	2	2	3	3	1	3	6	0	2	9	2	0	1	2	0	7	11
16	7	2	2	4	5	4	1	3	4	3	6	2	0	0	9	7	0	1
17	5	7	3	3	4	3	1	8	2	3	10	7	1	6	6	11	1	0

Now for the names of our types.

In [11]:

names = np.unique(types).tolist()
pd.DataFrame(names)

Out[11]:

	0
0	Bug
1	Dark
2	Dragon
3	Electric
4	Fairy
5	Fighting
6	Fire
7	Flying
8	Ghost
9	Grass
10	Ground
11	Ice
12	Normal
13	Poison
14	Psychic
15	Rock
16	Steel
17	Water

Chord Diagram¶

Time to visualise the co-occurrence of types using a chord diagram. We are going to use a list of custom colours that represent the types.

In [12]:

colors = ["#A6B91A", "#705746", "#6F35FC", "#F7D02C", "#D685AD",
          "#C22E28", "#EE8130", "#A98FF3", "#735797", "#7AC74C",
          "#E2BF65", "#96D9D6", "#A8A77A", "#A33EA1", "#F95587",
          "#B6A136", "#B7B7CE", "#6390F0"];

In [13]:

names

Out[13]:

['Bug',
 'Dark',
 'Dragon',
 'Electric',
 'Fairy',
 'Fighting',
 'Fire',
 'Flying',
 'Ghost',
 'Grass',
 'Ground',
 'Ice',
 'Normal',
 'Poison',
 'Psychic',
 'Rock',
 'Steel',
 'Water']

Finally, we can put it all together.

In [14]:

Chord(matrix, names, colors=colors).show()

Chord Diagram

Chord Diagram with Names¶

Note

The following example uses a customised version of Chord that supports the presentation of additional information.

It would be nice to show a list of Pokémon names when hovering over co-occurring Pokémon types. To do this, we can make use of the optional details parameter.

Let's clean up our dataset by removing all instances of \n.

In [15]:

data = data.replace('\n','', regex=True)

Next, we'll create an empty multi-dimensional array with the same shape as our matrix.

In [16]:

details = np.empty((len(names),len(names)),dtype=object)

Now we can populate the details array with lists of Pokémon names in the correct positions.

In [17]:

for count_x, item_x in enumerate(names):
    for count_y, item_y in enumerate(names):
        details[count_x][count_y] = data[
            (data['type_1'].isin([item_x, item_y])) &
            (data['type_2'].isin([item_y, item_x]))]['name'].to_list()

details=pd.DataFrame(details).values.tolist()

Finally, we can put it all together but this time with the details matrix passed in.

In [18]:

Chord(
    matrix,
    names,
    colors=colors,
    details=details,
    credit=True
).show()

Chord Diagram

	resident_status	education_1989_revision	education_2003_revision	education_reporting_flag	month_of_death	sex	detail_age_type	detail_age	age_substitution_flag	age_recode_52	...	record_condition_18	record_condition_19	record_condition_20	race	bridged_race_flag	race_imputation_flag	race_recode_3	race_recode_5	hispanic_origin	hispanic_originrace_recode
0	1	NaN	3.0	1	1	M	1	84	NaN	42	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6
1	1	NaN	6.0	1	1	M	1	70	NaN	40	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6
2	1	NaN	3.0	1	1	F	1	91	NaN	44	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6
3	1	NaN	3.0	1	1	F	1	40	NaN	34	...	NaN	NaN	NaN	3	NaN	NaN	2	3	100	8
4	1	NaN	5.0	1	1	F	1	89	NaN	43	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
0	0	0	0	4	2	4	4	14	1	6	2	2	0	13	2	5	7	5
1	0	0	4	2	3	3	4	5	3	3	3	2	5	5	3	2	2	7
2	0	4	0	3	1	2	3	8	5	5	9	3	1	4	5	2	2	3
3	4	2	3	0	2	0	1	6	1	1	1	2	2	3	1	3	4	3
4	2	3	1	2	0	0	0	2	1	5	0	1	5	1	9	3	5	4
5	4	3	2	0	0	0	7	1	1	3	0	1	4	2	6	1	4	3
6	4	4	3	1	0	7	0	7	5	0	4	1	2	2	3	3	1	1
7	14	5	8	6	2	1	7	0	3	7	4	2	27	3	7	6	3	8
8	1	3	5	1	1	1	5	3	0	12	6	1	0	4	3	0	4	2
9	6	3	5	1	5	3	0	7	12	0	1	3	2	15	3	2	3	3
10	2	3	9	1	0	0	4	4	6	1	0	3	1	2	2	9	6	10
11	2	2	3	2	1	1	1	2	1	3	3	0	0	0	4	2	2	7
12	0	5	1	2	5	4	2	27	0	2	1	0	0	0	5	0	0	1
13	13	5	4	3	1	2	2	3	4	15	2	0	0	0	0	1	0	6
14	2	3	5	1	9	6	3	7	3	3	2	4	5	0	0	2	9	6
15	5	2	2	3	3	1	3	6	0	2	9	2	0	1	2	0	7	11
16	7	2	2	4	5	4	1	3	4	3	6	2	0	0	9	7	0	1
17	5	7	3	3	4	3	1	8	2	3	10	7	1	6	6	11	1	0

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19
0	0	465	26	258	276	3	339	62	144	5	58	76	9	57	63	277	1	547	55	35
1	465	0	114	223	56	2	183	211	190	2	27	20	7	37	66	205	0	203	30	22
2	26	114	0	125	0	0	19	195	61	1	0	0	14	1	8	30	1	3	1	2
3	258	223	125	0	180	11	576	299	166	9	11	78	84	27	484	109	4	113	8	17
4	276	56	0	180	0	1	381	8	10	4	13	36	11	105	47	18	1	414	4	9
5	3	2	0	11	1	0	7	5	0	2	6	1	15	1	0	0	0	1	0	0
6	339	183	19	576	381	7	0	121	99	27	175	84	106	175	603	102	5	554	118	34
7	62	211	195	299	8	5	121	0	149	3	1	1	32	6	52	58	3	7	0	3
8	144	190	61	166	10	0	99	149	0	0	0	53	11	19	64	85	0	63	3	2
9	5	2	1	9	4	2	27	3	0	0	3	0	0	0	9	0	0	3	1	0
10	58	27	0	11	13	6	175	1	0	3	0	1	6	3	30	0	0	21	59	8
11	76	20	0	78	36	1	84	1	53	0	1	0	3	91	15	95	1	291	1	1
12	9	7	14	84	11	15	106	32	11	0	6	3	0	3	61	2	1	5	1	3
13	57	37	1	27	105	1	175	6	19	0	3	91	3	0	24	47	0	242	3	2
14	63	66	8	484	47	0	603	52	64	9	30	15	61	24	0	31	3	64	26	12
15	277	205	30	109	18	0	102	58	85	0	0	95	2	47	31	0	0	211	2	1
16	1	0	1	4	1	0	5	3	0	0	0	1	1	0	3	0	0	1	0	0
17	547	203	3	113	414	1	554	7	63	3	21	291	5	242	64	211	1	0	24	7
18	55	30	1	8	4	0	118	0	3	1	59	1	1	3	26	2	0	24	0	3
19	35	22	2	17	9	0	34	3	2	0	8	1	3	2	12	1	0	7	3	0

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
0	0	0	0	4	2	4	4	14	1	6	2	2	0	13	2	5	7	5
1	0	0	4	2	3	3	4	5	3	3	3	2	5	5	3	2	2	7
2	0	4	0	3	1	2	3	8	5	5	9	3	1	4	5	2	2	3
3	4	2	3	0	2	0	1	6	1	1	1	2	2	3	1	3	4	3
4	2	3	1	2	0	0	0	2	1	5	0	1	5	1	9	3	5	4
5	4	3	2	0	0	0	7	1	1	3	0	1	4	2	6	1	4	3
6	4	4	3	1	0	7	0	7	5	0	4	1	2	2	3	3	1	1
7	14	5	8	6	2	1	7	0	3	7	4	2	27	3	7	6	3	8
8	1	3	5	1	1	1	5	3	0	12	6	1	0	4	3	0	4	2
9	6	3	5	1	5	3	0	7	12	0	1	3	2	15	3	2	3	3
10	2	3	9	1	0	0	4	4	6	1	0	3	1	2	2	9	6	10
11	2	2	3	2	1	1	1	2	1	3	3	0	0	0	4	2	2	7
12	0	5	1	2	5	4	2	27	0	2	1	0	0	0	5	0	0	1
13	13	5	4	3	1	2	2	3	4	15	2	0	0	0	0	1	0	6
14	2	3	5	1	9	6	3	7	3	3	2	4	5	0	0	2	9	6
15	5	2	2	3	3	1	3	6	0	2	9	2	0	1	2	0	7	11
16	7	2	2	4	5	4	1	3	4	3	6	2	0	0	9	7	0	1
17	5	7	3	3	4	3	1	8	2	3	10	7	1	6	6	11	1	0

	resident_status	education_1989_revision	education_2003_revision	education_reporting_flag	month_of_death	sex	detail_age_type	detail_age	age_substitution_flag	age_recode_52	...	record_condition_18	record_condition_19	record_condition_20	race	bridged_race_flag	race_imputation_flag	race_recode_3	race_recode_5	hispanic_origin	hispanic_originrace_recode
0	1	NaN	3.0	1	1	M	1	84	NaN	42	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6
1	1	NaN	6.0	1	1	M	1	70	NaN	40	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6
2	1	NaN	3.0	1	1	F	1	91	NaN	44	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6
3	1	NaN	3.0	1	1	F	1	40	NaN	34	...	NaN	NaN	NaN	3	NaN	NaN	2	3	100	8
4	1	NaN	5.0	1	1	F	1	89	NaN	43	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
0	0	0	0	4	2	4	4	14	1	6	2	2	0	13	2	5	7	5
1	0	0	4	2	3	3	4	5	3	3	3	2	5	5	3	2	2	7
2	0	4	0	3	1	2	3	8	5	5	9	3	1	4	5	2	2	3
3	4	2	3	0	2	0	1	6	1	1	1	2	2	3	1	3	4	3
4	2	3	1	2	0	0	0	2	1	5	0	1	5	1	9	3	5	4
5	4	3	2	0	0	0	7	1	1	3	0	1	4	2	6	1	4	3
6	4	4	3	1	0	7	0	7	5	0	4	1	2	2	3	3	1	1
7	14	5	8	6	2	1	7	0	3	7	4	2	27	3	7	6	3	8
8	1	3	5	1	1	1	5	3	0	12	6	1	0	4	3	0	4	2
9	6	3	5	1	5	3	0	7	12	0	1	3	2	15	3	2	3	3
10	2	3	9	1	0	0	4	4	6	1	0	3	1	2	2	9	6	10
11	2	2	3	2	1	1	1	2	1	3	3	0	0	0	4	2	2	7
12	0	5	1	2	5	4	2	27	0	2	1	0	0	0	5	0	0	1
13	13	5	4	3	1	2	2	3	4	15	2	0	0	0	0	1	0	6
14	2	3	5	1	9	6	3	7	3	3	2	4	5	0	0	2	9	6
15	5	2	2	3	3	1	3	6	0	2	9	2	0	1	2	0	7	11
16	7	2	2	4	5	4	1	3	4	3	6	2	0	0	9	7	0	1
17	5	7	3	3	4	3	1	8	2	3	10	7	1	6	6	11	1	0

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19
0	0	465	26	258	276	3	339	62	144	5	58	76	9	57	63	277	1	547	55	35
1	465	0	114	223	56	2	183	211	190	2	27	20	7	37	66	205	0	203	30	22
2	26	114	0	125	0	0	19	195	61	1	0	0	14	1	8	30	1	3	1	2
3	258	223	125	0	180	11	576	299	166	9	11	78	84	27	484	109	4	113	8	17
4	276	56	0	180	0	1	381	8	10	4	13	36	11	105	47	18	1	414	4	9
5	3	2	0	11	1	0	7	5	0	2	6	1	15	1	0	0	0	1	0	0
6	339	183	19	576	381	7	0	121	99	27	175	84	106	175	603	102	5	554	118	34
7	62	211	195	299	8	5	121	0	149	3	1	1	32	6	52	58	3	7	0	3
8	144	190	61	166	10	0	99	149	0	0	0	53	11	19	64	85	0	63	3	2
9	5	2	1	9	4	2	27	3	0	0	3	0	0	0	9	0	0	3	1	0
10	58	27	0	11	13	6	175	1	0	3	0	1	6	3	30	0	0	21	59	8
11	76	20	0	78	36	1	84	1	53	0	1	0	3	91	15	95	1	291	1	1
12	9	7	14	84	11	15	106	32	11	0	6	3	0	3	61	2	1	5	1	3
13	57	37	1	27	105	1	175	6	19	0	3	91	3	0	24	47	0	242	3	2
14	63	66	8	484	47	0	603	52	64	9	30	15	61	24	0	31	3	64	26	12
15	277	205	30	109	18	0	102	58	85	0	0	95	2	47	31	0	0	211	2	1
16	1	0	1	4	1	0	5	3	0	0	0	1	1	0	3	0	0	1	0	0
17	547	203	3	113	414	1	554	7	63	3	21	291	5	242	64	211	1	0	24	7
18	55	30	1	8	4	0	118	0	3	1	59	1	1	3	26	2	0	24	0	3
19	35	22	2	17	9	0	34	3	2	0	8	1	3	2	12	1	0	7	3	0

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
0	0	0	0	4	2	4	4	14	1	6	2	2	0	13	2	5	7	5
1	0	0	4	2	3	3	4	5	3	3	3	2	5	5	3	2	2	7
2	0	4	0	3	1	2	3	8	5	5	9	3	1	4	5	2	2	3
3	4	2	3	0	2	0	1	6	1	1	1	2	2	3	1	3	4	3
4	2	3	1	2	0	0	0	2	1	5	0	1	5	1	9	3	5	4
5	4	3	2	0	0	0	7	1	1	3	0	1	4	2	6	1	4	3
6	4	4	3	1	0	7	0	7	5	0	4	1	2	2	3	3	1	1
7	14	5	8	6	2	1	7	0	3	7	4	2	27	3	7	6	3	8
8	1	3	5	1	1	1	5	3	0	12	6	1	0	4	3	0	4	2
9	6	3	5	1	5	3	0	7	12	0	1	3	2	15	3	2	3	3
10	2	3	9	1	0	0	4	4	6	1	0	3	1	2	2	9	6	10
11	2	2	3	2	1	1	1	2	1	3	3	0	0	0	4	2	2	7
12	0	5	1	2	5	4	2	27	0	2	1	0	0	0	5	0	0	1
13	13	5	4	3	1	2	2	3	4	15	2	0	0	0	0	1	0	6
14	2	3	5	1	9	6	3	7	3	3	2	4	5	0	0	2	9	6
15	5	2	2	3	3	1	3	6	0	2	9	2	0	1	2	0	7	11
16	7	2	2	4	5	4	1	3	4	3	6	2	0	0	9	7	0	1
17	5	7	3	3	4	3	1	8	2	3	10	7	1	6	6	11	1	0

	resident_status	education_1989_revision	education_2003_revision	education_reporting_flag	month_of_death	sex	detail_age_type	detail_age	age_substitution_flag	age_recode_52	...	record_condition_18	record_condition_19	record_condition_20	race	bridged_race_flag	race_imputation_flag	race_recode_3	race_recode_5	hispanic_origin	hispanic_originrace_recode
0	1	NaN	3.0	1	1	M	1	84	NaN	42	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6
1	1	NaN	6.0	1	1	M	1	70	NaN	40	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6
2	1	NaN	3.0	1	1	F	1	91	NaN	44	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6
3	1	NaN	3.0	1	1	F	1	40	NaN	34	...	NaN	NaN	NaN	3	NaN	NaN	2	3	100	8
4	1	NaN	5.0	1	1	F	1	89	NaN	43	...	NaN	NaN	NaN	1	NaN	NaN	1	1	100	6

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
0	0	0	0	4	2	4	4	14	1	6	2	2	0	13	2	5	7	5
1	0	0	4	2	3	3	4	5	3	3	3	2	5	5	3	2	2	7
2	0	4	0	3	1	2	3	8	5	5	9	3	1	4	5	2	2	3
3	4	2	3	0	2	0	1	6	1	1	1	2	2	3	1	3	4	3
4	2	3	1	2	0	0	0	2	1	5	0	1	5	1	9	3	5	4
5	4	3	2	0	0	0	7	1	1	3	0	1	4	2	6	1	4	3
6	4	4	3	1	0	7	0	7	5	0	4	1	2	2	3	3	1	1
7	14	5	8	6	2	1	7	0	3	7	4	2	27	3	7	6	3	8
8	1	3	5	1	1	1	5	3	0	12	6	1	0	4	3	0	4	2
9	6	3	5	1	5	3	0	7	12	0	1	3	2	15	3	2	3	3
10	2	3	9	1	0	0	4	4	6	1	0	3	1	2	2	9	6	10
11	2	2	3	2	1	1	1	2	1	3	3	0	0	0	4	2	2	7
12	0	5	1	2	5	4	2	27	0	2	1	0	0	0	5	0	0	1
13	13	5	4	3	1	2	2	3	4	15	2	0	0	0	0	1	0	6
14	2	3	5	1	9	6	3	7	3	3	2	4	5	0	0	2	9	6
15	5	2	2	3	3	1	3	6	0	2	9	2	0	1	2	0	7	11
16	7	2	2	4	5	4	1	3	4	3	6	2	0	0	9	7	0	1
17	5	7	3	3	4	3	1	8	2	3	10	7	1	6	6	11	1	0

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19
0	0	465	26	258	276	3	339	62	144	5	58	76	9	57	63	277	1	547	55	35
1	465	0	114	223	56	2	183	211	190	2	27	20	7	37	66	205	0	203	30	22
2	26	114	0	125	0	0	19	195	61	1	0	0	14	1	8	30	1	3	1	2
3	258	223	125	0	180	11	576	299	166	9	11	78	84	27	484	109	4	113	8	17
4	276	56	0	180	0	1	381	8	10	4	13	36	11	105	47	18	1	414	4	9
5	3	2	0	11	1	0	7	5	0	2	6	1	15	1	0	0	0	1	0	0
6	339	183	19	576	381	7	0	121	99	27	175	84	106	175	603	102	5	554	118	34
7	62	211	195	299	8	5	121	0	149	3	1	1	32	6	52	58	3	7	0	3
8	144	190	61	166	10	0	99	149	0	0	0	53	11	19	64	85	0	63	3	2
9	5	2	1	9	4	2	27	3	0	0	3	0	0	0	9	0	0	3	1	0
10	58	27	0	11	13	6	175	1	0	3	0	1	6	3	30	0	0	21	59	8
11	76	20	0	78	36	1	84	1	53	0	1	0	3	91	15	95	1	291	1	1
12	9	7	14	84	11	15	106	32	11	0	6	3	0	3	61	2	1	5	1	3
13	57	37	1	27	105	1	175	6	19	0	3	91	3	0	24	47	0	242	3	2
14	63	66	8	484	47	0	603	52	64	9	30	15	61	24	0	31	3	64	26	12
15	277	205	30	109	18	0	102	58	85	0	0	95	2	47	31	0	0	211	2	1
16	1	0	1	4	1	0	5	3	0	0	0	1	1	0	3	0	0	1	0	0
17	547	203	3	113	414	1	554	7	63	3	21	291	5	242	64	211	1	0	24	7
18	55	30	1	8	4	0	118	0	3	1	59	1	1	3	26	2	0	24	0	3
19	35	22	2	17	9	0	34	3	2	0	8	1	3	2	12	1	0	7	3	0

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
0	0	0	0	4	2	4	4	14	1	6	2	2	0	13	2	5	7	5
1	0	0	4	2	3	3	4	5	3	3	3	2	5	5	3	2	2	7
2	0	4	0	3	1	2	3	8	5	5	9	3	1	4	5	2	2	3
3	4	2	3	0	2	0	1	6	1	1	1	2	2	3	1	3	4	3
4	2	3	1	2	0	0	0	2	1	5	0	1	5	1	9	3	5	4
5	4	3	2	0	0	0	7	1	1	3	0	1	4	2	6	1	4	3
6	4	4	3	1	0	7	0	7	5	0	4	1	2	2	3	3	1	1
7	14	5	8	6	2	1	7	0	3	7	4	2	27	3	7	6	3	8
8	1	3	5	1	1	1	5	3	0	12	6	1	0	4	3	0	4	2
9	6	3	5	1	5	3	0	7	12	0	1	3	2	15	3	2	3	3
10	2	3	9	1	0	0	4	4	6	1	0	3	1	2	2	9	6	10
11	2	2	3	2	1	1	1	2	1	3	3	0	0	0	4	2	2	7
12	0	5	1	2	5	4	2	27	0	2	1	0	0	0	5	0	0	1
13	13	5	4	3	1	2	2	3	4	15	2	0	0	0	0	1	0	6
14	2	3	5	1	9	6	3	7	3	3	2	4	5	0	0	2	9	6
15	5	2	2	3	3	1	3	6	0	2	9	2	0	1	2	0	7	11
16	7	2	2	4	5	4	1	3	4	3	6	2	0	0	9	7	0	1
17	5	7	3	3	4	3	1	8	2	3	10	7	1	6	6	11	1	0