Ludwig Passini - Cafe Greco. Depicted is one of the oldest cafes in Rome, first opened in 1760.

Discrete-event simulation

In a discrete-event simulation we model a system as a sequence of events that occur at discrete times. For example, in a grocery queue, the events might be: (1) customer lines up at the queue, (2) customer reaches cashier, and (3) completes the transaction. The entire system is modeled via events and no state change occurs between events.

Discrete event simulations have a large number of applications in logistics, operations and many others. They can be a useful addition in a data scientist's toolbelt.

Simpy

Simpy is a Python package for discrete-event simulation. The package provides three main constructs: events, processes and resources. We will discuss resources in the Coffee Shop simulation, below we illustrate the first two constructs.

Note: The basic Simpy components (events, processes and resources) can be combined to model rich dynamics.

Processes model components of a system by emitting events. Processes are defined by a Python generator in Simpy. A generator allows for the temporary suspension of a process, and it makes it very convenient to model various components. Here is an example:

import simpy

def worker(env, worker_id, time_to_complete):
    """A worker finishes a part every time_to_complete minutes"""
    i = 1
    while True:
        yield env.timeout(time_to_complete)
        print(f"worker {worker_id} finished {i} parts at {env.now}")
        i += 1
        
env = simpy.Environment()
env.process(worker(env, worker_id=1, time_to_complete=20))
env.process(worker(env, worker_id=2, time_to_complete=30))
env.run(until=100)

worker 1 finished 1 parts at 20
worker 2 finished 1 parts at 30
worker 1 finished 2 parts at 40
worker 2 finished 2 parts at 60
worker 1 finished 3 parts at 60
worker 1 finished 4 parts at 80
worker 2 finished 3 parts at 90

Above we create two worker processes. A worker simply takes time_to_complete minutes to complete a part. The "waiting" part is determined by a simpy Timeout event, which allows for simulation time to pass. When the timeout is complete (the event has been processed), the worker process is resumed by the environment and the entire procedure repeats.Note that the two workers are running concurrently.

Note: The Simpy environment keeps track of events in an event queue and will resume the appropriate process once an event has been processed.

Now, onto the coffee shop simulation!

Coffee Shop

In the coffee shop simulation, we model the process of ordering and preparing a drink. Each customer orders n_cups of coffee. For simplicity, we model the number of cups only, and not the details of the order. There will be no caramel macchiato with chocolate drizzle available in our coffee shop! Most customers buy one or two cups of coffee, occasionally up to four.

We also sample the times between customer arrivals (interarrival) from an exponential distribution. This is a common choice as it assumes customers are arriving independently of each other. Below is an example using a mean interarrival time of 2 minutes with 20 customers.

import random
from functools import partial
import numpy as np
import matplotlib.pyplot as plt

def pick_n_cups():
    return random.choices(range(1, 5), [0.6, 0.25, 0.05, 0.05])[0]

def interarrival(mean):
    assert mean > 0
    return random.expovariate(1 / mean)

mean_delta = 2  # minutes
n_customers = 20
arrival_deltas = [interarrival(mean_delta) for _ in range(n_customers)]
n_cups = [pick_n_cups() for _ in range(n_customers)]
arrival_times = np.cumsum(arrival_deltas)

#collapse
plt.figure(figsize=(10, 4))
ax = plt.gca()
ax.bar(arrival_times, n_cups, width=0.2)
ax.set_yticks(range(1, 5))
ax.set_xlabel("Time [minutes]")
ax.set_ylabel("Number of cups")
ax.set_title(f"Arrival times with mean = {mean_delta} minutes");

We see that it takes about 40 minutes for 20 customers to arrive, as we would expect with a mean of 2 minutes.

Barristas and resources

Let's start with a simplified model where we only have barristas, so a customer can directly request coffee from them without going through checkout. Free coffee!

We use a Simpy Resource to model barristas. A resource is typically shared between multiple processes, and they queue to use it. A resource has a capacity - in this case this determines the number of barristas. Other types of resources are also available: for example, containers can be used to model a continuous quantity.

from collections import namedtuple

Order = namedtuple("Order", ["id", "n_cups"])

def time_per_cup(avg_time=2):
    return max(1, random.normalvariate(avg_time, 0.5))

def prepare_coffee(env, order, barrista):
    print(f"begin order {order.id}", round(env.now, 3))
    with barrista.request() as req:
        yield req  # wait until a barrista is available
        prepare_time = order.n_cups * time_per_cup()
        yield env.timeout(prepare_time)  # preparing coffee
        
    print(f"order {order.id} ready!", round(env.now, 3))
    
def generate_customers(env, barrista, mean_delta=2):
    i = 0
    while True:
        order = Order(i, pick_n_cups())
        yield env.timeout(interarrival(mean_delta))
        env.process(prepare_coffee(env, order, barrista))
        i += 1

Each order is a namedtuple, which contains the order id (so we can track it throughout the simulation), as well as the n_cups which impacts the time it takes to prepare the order. We assume the order prep time is linear with n_cups, with same variability determined by time_per_cup.

The simulation currently consists of two processes. The main process is prepare_coffee which models how a coffee is prepared. We first request the barrista resource, which means we wait until a barrista is available, and then prepare the order based on n_cups. When we exist the barrista.request() context, the barrista is released by the current process, and becomes available to fulfull other orders.

The generate_customers resource generates orders based a mean_delta value using the interarrival distribution we saw before.

random.seed(0)
env = simpy.Environment()
barrista = simpy.Resource(env, capacity=1)
env.process(generate_customers(env, barrista))
env.run(until=20)

begin order 0 2.837
begin order 1 3.437
begin order 2 4.159
begin order 3 5.911
order 0 ready! 6.87
begin order 4 7.316
begin order 5 7.892
order 1 ready! 8.103
order 2 ready! 10.898
order 3 ready! 12.871
begin order 6 16.016
order 4 ready! 18.436

Refactoring, refactoring

Based on the simple run above, we see that a single barrista seems to be struggling with the flood of clients during the busy period. Let's refactor the simulation code so it is easier to track customer wait times.

We create a class CoffeeSimBase which implements the simulation logic. Inside the class, we keep referenences to the environment, the barrista resources, as well as a dictionary records which marks each time an order begins (+1), and is completed (-1). In this way, we can calculate the number of orders waiting at any one time by computing a cumulative sum over the records. We do not yet make any changes to the simulation logic discussed above.

ORDER_BEGIN = 1
ORDER_READY = -1

class CoffeeSimBase:
    """Base coffee simulation - with barristas only."""
    def __init__(self, env, n_barristas, to_log=False):
        self.env = env
        self.barrista = simpy.Resource(env, capacity=n_barristas)
        self.records = {}  # for saving data
        self.to_log = to_log
        
    def record_order_begin(self, order):
        begin_time = self.env.now
        self.records[begin_time] = ORDER_BEGIN
        if self.to_log: print(f"begin order {order.id}", round(self.env.now, 3))
        
    def record_order_ready(self, order):
        ready_time = self.env.now
        self.records[ready_time] = ORDER_READY
        if self.to_log: print(f"order {order.id} ready!", round(ready_time, 3))
        
    def request_barrista(self, order):
        with self.barrista.request() as req:
            yield req  # wait until a barrista is available
            prepare_time = order.n_cups * time_per_cup()
            yield self.env.timeout(prepare_time)
        
    def prepare_coffee(self, order):
        self.record_order_begin(order)
        yield self.env.process(self.request_barrista(order))
        self.record_order_ready(order)

    def generate_customers(self, mean_delta=2):
        i = 0
        while True:
            order = Order(i, pick_n_cups())
            yield self.env.timeout(interarrival(mean_delta))
            self.env.process(self.prepare_coffee(order))
            i += 1
     
    def run(self, mean_delta, until):
        self.env.process(self.generate_customers(mean_delta))
        self.env.run(until=until)
        
    @classmethod
    def from_new_env(cls, *args, **kwargs):
        env = simpy.Environment()
        return cls(env, *args, **kwargs)

coffee_sim = CoffeeSimBase.from_new_env(n_barristas=1, to_log=True)
coffee_sim.run(2, 20)

begin order 0 3.27
begin order 1 3.298
begin order 2 5.504
order 0 ready! 5.583
begin order 3 6.865
order 1 ready! 8.0
order 2 ready! 10.775
begin order 4 10.953
begin order 5 11.724
order 3 ready! 13.246
order 4 ready! 14.499
order 5 ready! 16.89
begin order 6 17.125
begin order 7 18.68
order 6 ready! 19.261
begin order 8 19.652

Below we add some helper functions to collect data from simulation runs and plot them.

#collapse
import pandas as pd
import seaborn as sns
from itertools import islice

def records2series(ledger):
    """Convert the coffee ledger into a series."""
    series = pd.Series(ledger).cumsum()
    assert series.min() >= 0
    return series

def simulate_many(sim_factory, mean_delta=2, until=60):
    while True:
        sim = sim_factory()
        sim.run(mean_delta=mean_delta, until=until)
        yield sim

def collect_records(sim_iterator, n=10):
    sim_recors = [sim.records for sim in islice(sim_iterator, n)]        
    return [records2series(r) for r in sim_recors]

def lineplot_many(records, ax=None, **kwargs):
    ax = ax or plt.gca()
    for r in records:
        r.plot(ax=ax, color="steelblue", **kwargs)
    ax.set_xlabel("Time [minutes]")
    ax.set_ylabel("queue size")
    return ax

sim_one = partial(CoffeeSimBase.from_new_env, n_barristas=1)
sim_two = partial(CoffeeSimBase.from_new_env, n_barristas=2)

#collapse
records_one = collect_records(simulate_many(sim_one, until=90))
records_two = collect_records(simulate_many(sim_two, until=90))

fig, (ax0, ax1) = plt.subplots(nrows=1, ncols=2, figsize=(12, 4), sharey=True)
ax0 = lineplot_many(records_one, ax=ax0, alpha=0.8)
ax1 = lineplot_many(records_two, ax=ax1, alpha=0.8)
ax0.set_title("barristas=1")
ax1.set_title("barristas=2");

With one barrista, the queue size increases steadily over time, in some cases to more than 20 people after one hour.

With two barristas, the queue might occasionally get longer, but in most cases it is manageable.

Have a little patience

So far we have assumed that each customer has infinite patience and will wait until the end of time for their drink. In practice, a very long queue might discourage some customers. We are going to model two populations, customers who have near-infinite patience, and customers with relatively short patience, e.g. 5 minutes. If the latter group's order is not complete within this period, they will exit the queue.

To model this, we will use two Simpy features. The first is the ability to wait for multiple events to complete. The following will complete when either event a or b are processed:

res = yield event_a | event_b  # wait for a or b

As expected, there is a related construct of waiting for both events a and b:

res = yield event_a & event_b  # wait for a and b

but we do not use this in our simulation.

The second feature we need is the ability to interrupt a process, by triggering an interrupt exception. Once an interrupt is triggered, it is passed to the process generator - in our case request_barrista. So we need to modify it to handle the exception.

Here is an updated coffee simulation class:

T_IMPATIENT = 5
T_PATIENT = 1000

def get_patience(p_patient=0.7):
    """Some people have patience = 5 minutes, and 
    others, nearly infinite patience."""
    return T_PATIENT if random.random() < p_patient else T_IMPATIENT

class CoffeeSimPatience(CoffeeSimBase):
    def __init__(self, env, n_barristas, to_log=False):
        super().__init__(env, n_barristas, to_log)
        self.orders_left = []
        
    def request_barrista(self, order):
        with self.barrista.request() as req:
            try:
                yield req  # wait until a barrista is available
                prepare_time = order.n_cups * time_per_cup()
                yield self.env.timeout(prepare_time)
            except simpy.Interrupt as i:
                return
            
    def record_order_left(self, order):
        """Keep track of orders where customer ran out of patience."""
        leave_time = self.env.now
        if self.to_log: print(f"leaving order {order.id} at", round(leave_time, 2))
        self.orders_left.append(order)
        
    def prepare_coffee(self, order):
        self.record_order_begin(order)
        patience = self.env.timeout(delay=get_patience())
        barrista_req = self.env.process(self.request_barrista(order))
        res = yield patience | barrista_req
        if barrista_req not in res:  # ran out of patience
            barrista_req.interrupt()
            self.record_order_left(order)
            return
        self.record_order_ready(order)
        
        
def collect_orders_counts(sim_iterator, n=10):
    """Collect tuples of (orders_lost, orders_complete)"""
    order_counts = []
    for sim in islice(sim_iterator, n):
        n_orders_lost = len(sim.orders_left)
        n_orders_complete = sum(1 for v in sim.records.values() if v == ORDER_BEGIN)
        order_counts.append((n_orders_lost, n_orders_complete))
    return order_counts

If the customer runs out of patience, the barrista_req process (which hasn't completed yet) is interrupted. We have modified the request_barrista generator function to handle this interrupt; in that case it will simply return, thereby releasing the barrista resource.

sim_patience_one = partial(CoffeeSimPatience.from_new_env, n_barristas=1)
sim_patience_two = partial(CoffeeSimPatience.from_new_env, n_barristas=2)

records_one = collect_orders_counts(simulate_many(sim_patience_one, until=90), n=50)
records_two = collect_orders_counts(simulate_many(sim_patience_two, until=90), n=50)

print("barristas=1", records_one[:5])
print("barristas=2", records_two[:5])

barristas=1 [(10, 46), (12, 55), (14, 52), (15, 50), (8, 35)]
barristas=2 [(2, 39), (8, 54), (6, 46), (2, 30), (4, 37)]

#collapse
records_one = (pd.DataFrame(records_one, columns=["orders_lost", "orders_complete"])
               .assign(barrista=1))
records_two = (pd.DataFrame(records_two, columns=["orders_lost", "orders_complete"])
               .assign(barrista=2))
records = pd.concat([records_one, records_two], ignore_index=True)
records["prop_lost"] = records["orders_lost"] / (records["orders_complete"] + records["orders_lost"])
ax = sns.boxplot(data=records, x="barrista", y="prop_lost")
ax.set_ylabel("Proportion of orders lost")
ax.set_xlabel("Number of barristas");
ax.set_title("Orders lost with barristas only");

Adding a cashier

Most busy coffee shops have a separate cashier accepting orders. This effectively creates two customer queues, one for placing an order, and one for collecting drinks from the barrista. For our final simulation, we will add a cashier.

Importantly, we are going to assume that a customer can only run out of patience while waiting in the cashier queue. Once they've paid, the assumption is they would wait to collect their drink.

def cashier_time(avg_time=1.5):
    """Time it takes cashier to collect order (in minutes)"""
    return max(0.5, random.normalvariate(avg_time, 0.5)) 

class CoffeeSimCashier(CoffeeSimPatience):
    def __init__(self, env, n_barristas, n_cashiers=1, to_log=False):
        super().__init__(env, n_barristas, to_log)
        self.cashier = simpy.Resource(env, capacity=n_cashiers)
        
    def request_cashier(self, order):
        with self.cashier.request() as req:
            try:
                yield req  # wait until a cashier is available
                yield self.env.timeout(cashier_time())
            except simpy.Interrupt as i:
                return
        
    def prepare_coffee(self, order):
        self.record_order_begin(order)
        patience = self.env.timeout(delay=get_patience())
        cashier_req = self.env.process(self.request_cashier(order))
        res = yield patience | cashier_req
        if cashier_req not in res:  # ran out of patience
            cashier_req.interrupt()
            self.record_order_left(order)
            return
        
        yield self.env.process(self.request_barrista(order))
        self.record_order_ready(order)

The request_cashier process is similar to the request_barrista, where it needs to handle the Interrupt when a customer leaves.

sim_patience_one = partial(CoffeeSimCashier.from_new_env, n_barristas=1)
sim_patience_two = partial(CoffeeSimCashier.from_new_env, n_barristas=2)

records_one = collect_orders_counts(simulate_many(sim_patience_one, until=90), n=100)
records_two = collect_orders_counts(simulate_many(sim_patience_two, until=90), n=100)
print("barristas=1", records_one[:5])
print("barristas=2", records_two[:5])

barristas=1 [(2, 49), (0, 36), (0, 42), (0, 44), (3, 38)]
barristas=2 [(4, 49), (0, 37), (0, 41), (2, 46), (2, 50)]

#collapse
records_one = (pd.DataFrame(records_one, columns=["orders_lost", "orders_complete"])
               .assign(barrista=1))
records_two = (pd.DataFrame(records_two, columns=["orders_lost", "orders_complete"])
               .assign(barrista=2))
records = pd.concat([records_one, records_two], ignore_index=True)
records["prop_lost"] = records["orders_lost"] / (records["orders_complete"] + records["orders_lost"])
ax = sns.boxplot(data=records, x="barrista", y="prop_lost")
ax.set_ylabel("Proportion of orders lost")
ax.set_xlabel("Number of barristas")
ax.set_title("Orders lost with cashier and barristas");

We see that the proportion of orders lost is significantly reduced when a cashier is introduced to the simulation. This is because we assumed a customer wouldn't leave once they've paid for their drink. It usually takes longer to prepare the drink rather than to pay for it, so customers are spending more time in the second queue, waiting for the barrista.

In practice, the reduction in orders lost is not as dramatic because a cashier typically wouldn't not let the barrista queue get too long. In fact, they might jump in and help out some of the barristas, speeding up the barrista queue, but slowing down the cashier queue. Still, having a dedicated cashier impacts the ordering dynamics.

Further exploration

There are many ways in which this simple model can be extended. One interesting idea is to model order mistakes. Even in the best of coffee shops, mistakes can occur, where there is a misunderstanding between the customer and the cashier / barrista about the order details. Even though these mistakes are typically rare, I've noticed they can cause a substantial disruption: often a discussion is needed to clarify the issue, and the order might be redone, introducing a significant delay into the system.

This issue could be modeled by a mixture distribution, where most drinks are prepared according to plan, but a small number of drinks take significantly longer to prepare. Furthermore, this delay can be modeled in both the cashier process and the barrista process.

Conclusion

We used discrete-event simulation to model a coffee shop. While the model is relatively simple, we discovered insights about the structure of the coffee shop queues as they relate to lost orders.

Simpy takes advantage of generators in Python to model rich process dynamics. This is because generators have a key ability to suspend execution and then resume at a later point. This has important applications in other areas such as asynchronous programming.

#collapse
import pickle
from collections import defaultdict
from itertools import product
from pathlib import Path

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from tqdm import tqdm

from imblearn.datasets import fetch_datasets
from imblearn.ensemble import BalancedRandomForestClassifier, BalancedBaggingClassifier
from imblearn.over_sampling import SMOTE, RandomOverSampler
from imblearn.pipeline import make_pipeline

from sklearn.datasets import make_blobs
from sklearn.model_selection import cross_validate, cross_val_predict, StratifiedKFold
from sklearn.metrics import precision_recall_curve
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

Introduction

Imbalanced datasets occur in various fields and create unique challenges in model training as well as evaluation. In this post, we compare a few ensemble algorithms on imbalanced datasets. We delve deeper on a specific model: a bag of logistic regressions, and discuss why it might be a good choice for many imbalanced datasets.

Datasets

We use three datasets from different domains. All three are available from the UCI ML Database. An overview of their properties is provided below.

#collapse

abalone_fname = 'abalone.feather'

def fetch_abalone():
    """Download Abalone dataset from UCI archives and load in Pandas"""
    abalone_uci = 'http://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data'
    abalone = pd.read_csv(abalone_uci, header=None)
    abalone.columns = ['sex', 'length', 'diameter', 'height', 'whole_weight', 'shucked_weight', 
                       'viscera_weight', 'shell_weight', 'rings']
    abalone = pd.get_dummies(abalone, columns=['sex'])
    abalone.to_feather(abalone_fname)  # save local copy
    return abalone

def abalone_Xy(thr):
    """For a given threshold thr, all values above the threshold become
    the positive class in binary classification"""
    return {
        'data': abalone.drop(columns='rings'),
        'target': (abalone['rings'] >= thr).astype(np.int32),
        'DESCR': f'abalone_{thr}+'
    }

def get_stats(ds):
    """Extract statistics from a dataset."""
    n_examples, n_feats = ds['data'].shape
    is_positive = ds['target'] == 1
    return {
        'name': ds['DESCR'],
        'n_examples': n_examples,
        'n_feats': n_feats,
        'prop_positive': is_positive.mean(),
        'n_positive': is_positive.sum()
    }


if Path(abalone_fname).is_file():
    abalone = pd.read_feather(abalone_fname)
else:
    abalone = fetch_abalone()
    
datasets = fetch_datasets(filter_data=('ozone_level', 'coil_2000'))
thr = 16
datasets[f'abalone_{thr}+'] = abalone_Xy(thr=thr)

ds_info = pd.DataFrame(get_stats(ds) for ds in datasets.values())
ds_info

Ozone level

This dataset contains 7 years of daily metereological (wind speed, temperature) as well as air quality indicators for Houson, Texas. The goal is to predict "ozone days" - days when the ground-level ozone concentration exceeds a certain threshold, which can be harmful to human health.

Coil 2000

This dataset was used for the CoIL data mining competition. It is an insurance dataset where the goal is to predict which customers of a Dutch insurance company own an insurance policy covering a caravan. The features include socio-demographic data (derived from postal codes) and product ownership data. Importantly, all continuous features have been discretized in advance by the company.

Abalone

Abalone are molluscs and a type of marine snail. This dataset contains physical measurements of abalone (weight, size, length, sex) and the target variable is the number of rings (which is related to their age). The ground truth for the rings is obtained by cutting the shell and staining it, which is a time-consuming task, whereas the physical measurements are more easily obtained.

For the purposes of creating an imbalanced dataset, we take all abalone with rings >= 16 as the minority class at about 6%.

Experiments

Models

Dealing with imbalanced datasets often involves resampling. For example, we can under-sample the majority class, over-sample the minority class, or do a combination of the two. There are also methods that over-sample based on specific rules, which might involve generating synthetic examples. e.g., see the family of SMOTE methods in the imbalanced-learn library.

Bagging methods can be a particularly good fit for imbalanced datasets because resampling is already part of the algorithm. All we need to do is to modify the resampling procedure to introduce rebalancing of the dataset. This is already implemented for us in imblearn.ensemble. For example, a BalancedRandomForestClassifier(sampling_strategy=0.5) is similar to a regular Random Forest, but each tree will be fitted to a rebalanced dataset, where the ratio of minority- to majority-class samples is 0.5

The models we use are:

Baseline model: RandomOversampler followed by a single logistic regression, trained on the rebalanced dataset.
Balanced Random Forest (or a Bag of Trees)
Balanced Bag of Logistic Regressions. Similar to the balanced forest, but the base model is a logistic regression.

The rationale for this choice is that a balanced forest model might be able to capture non-linear relationships and interactions between variables, but the bag of logistic regressions might be more robust to overfitting in the imbalanced setting where label information is sparse.

The code below includes the model definitions:

def get_clfs(sampling_strategy, max_features=0.5):
    """Define ensemble classifiers for experiment.
    
    sampling_strategy: float or str
        Target ratio for minority / majority class when rebalancing.
        With sampling_strategy=0.5, there will be twice as many samples in the majority class
        after the rebalancing procedure.
    max_features: the proportion of features to draw at each resampling.
    """
    return {
        'single_logistic': make_pipeline(
            StandardScaler(),
            RandomOverSampler(sampling_strategy=sampling_strategy),
            LogisticRegression(penalty='l2')
        ),
        'logistic_bag': make_pipeline(
            StandardScaler(),
            BalancedBaggingClassifier(
                base_estimator=LogisticRegression(penalty='l2'),
                sampling_strategy=sampling_strategy,
                max_features=max_features, 
                n_estimators=20            
            )
        ),
        'balanced_forest': BalancedRandomForestClassifier(
            min_samples_split=10, 
            sampling_strategy=sampling_strategy,
            n_estimators=100,
            max_features=max_features)
    }

Evaluation

To evaluate the models, we will use 10-fold cross-validation, using a stratified split, that will ensure that each split receives roughly the same number of positive examples. This should produce more stable results for imbalanced dataset compared to a regular cross-validation based on a shuffled split.

We will examine the precision-recall curves obtained from the cross-validation. We also pick somewhat arbitrarily a recall threshold of 60%, and see which model will obtain better precision at that level.

#collapse
def run_exp(ds, clf, seed=42):
    """Given a dataset ds and classifier clf, run a cross-validation experiment."""
    X, y = ds['data'], ds['target']
    cv = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
    scores = cross_validate(clf, X, y, cv=cv, n_jobs=-1,
                            scoring=['balanced_accuracy', 'f1', 'precision', 'recall'])
    y_probs = cross_val_predict(clf, X, y, method='predict_proba', cv=cv, n_jobs=-1)[:, 1]
    scores['y_probs'] = y_probs
    return scores

def plot_precision_recall(y_true, y_probs, ax=None, **plot_kwargs):
    ax = ax or plt.gca()
    prec, recall, thr = precision_recall_curve(y_true, y_probs)
    ax.plot(recall, prec, **plot_kwargs)
    ax.set_ylabel('Precision', fontsize=12)
    ax.set_xlabel('Recall', fontsize=12)
    ax.set_ylim(0, 1)
    return ax

def collect_results(datasets, clfs):
    """Collect experiment results.
    
    The results shape is {<dataset_name>: {'classifier_name'}: result}
    """
    results = defaultdict(dict)
    for (ds_name, ds), (clf_name, clf) in tqdm(product(datasets.items(), clfs.items()),
                                               total=len(datasets) * len(clfs)):
        results[ds_name][clf_name] = run_exp(ds, clf)
    return results

results = collect_results(datasets, get_clfs(sampling_strategy=0.5))
with open('base_results.pickle', 'wb') as f:
    pickle.dump(results, f)

100%|██████████| 9/9 [00:53<00:00,  5.92s/it]

Results

Below are the plots of the precision-recall curves.

#collapse
fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(14, 3.5), sharey=True)
for i, (ds_name, ds_dict) in enumerate(results.items()):
    ax = axes[i]
    y_true = datasets[ds_name]['target']
    for clf_name, res in ds_dict.items():
        ax = plot_precision_recall(y_true, res['y_probs'], ax=ax, label=clf_name)
    ax.set_title(ds_name)
    ax.legend()
    ax.grid(True)

For the Ozone dataset, the bag of logistic regressions outperforms the balanced forest, at least for near the recall region of interest at recall=0.6. On the other hand, the balanced forest does slightly better on the Abalone dataset.

Both models do poorly on the Coil dataset. Sadly, one of the great challenges of our time, predicting caravan insurance policy, remains unsolved.

Note that the Ozone dataset includes 72 features, but it is the most imbalanced, at about 2.9%. The amount of signal available from the limited positive labels is likely quite sparse. This is possibly why the bag of logistic models does better than the balanced forest.

The Abalone dataset contains the smallest number of features (10). It is possible that the ability of the random forest to capture interactions and non-linear effects gave it a slight edge over the other model.

Bag of logistic regressions

The bag of logistic regressions performed quite well in this experiment but it might not be an obvious choice. So it is worth spending a moment to consider why it might work so well. We address a few questions about it below.

Isn't a bag of linear models simply a linear model?

It depends on the linear model, and how the individual models' predictions are combined in the bag. Indeed, for regression problems, a bag of linear regressions will produce another linear model. Logistic regression, however, includes a link function:

$$p_i(x) = \frac {1}{1+e^{-\mathbf{w x_i}}}$$

where $\mathbf{w}$ is the vector of weights fitted in the model, and $p_i$ is the predicted probability for example $x_i$.

In a BaggingClassifier, we are averaging the predicted probabilities of the base models - basically it uses the predict_proba method averages these together. For example, with two models, we obtain:

$$p_{i, bag}(x) = \frac{p_{i, 1} + p_{i, 2}}{2} = \frac{1}{2} [\frac {1}{1+e^{-\mathbf{w_1 x_i}}} + \frac {1}{1+e^{-\mathbf{w_2 x_i}}}]$$

This combination of the $p_i$ value will not necessarily produce a linear decision boundary as long as the weights $\mathbf{w_i}$ are sufficiently different. We can generate a few toy 2D datasets to illustrate this.

#collapse
def make_imbalanced_blobs(n_positive, n_negative=400, randon_state=0):
    """Creates a synthetic dataset with four blobs, where the blob at
    (0, 0) is the negative class.
    
    By setting n_positive to a small number, we can create an imbalanced dataset.
    """
    centers = [(0, 0), (5, 5), (5, -5), (-5, 5)]
    total = n_negative * len(centers)
    assert n_positive + n_negative <= total, f'n_positive + n_negative exceeds total {total}'
    X, y_blob = make_blobs(n_samples=n_negative * len(centers), centers=centers, 
                           shuffle=False, random_state=randon_state)
    X_center = X[y_blob == 0]  # negative class
    X_side   = X[y_blob != 0]  # positive class before resampling
    sample = np.random.choice(len(X_side), size=n_positive, replace=False)
    X_side = X_side[sample]

    X = np.concatenate([X_center, X_side])
    y = np.concatenate([np.zeros(len(X_center), dtype=int), 
                        np.ones(len(X_side),    dtype=int)])
    return X, y

def plot_blobs_with_boundaries(clfs, X, y):
    """Plots the decision boundaries for classifiers clfs, given 
    a blobs dataset. One plot for each classifier is created.
    """
    for clf in clfs.values(): clf.fit(X, y)

    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.1),
                         np.arange(y_min, y_max, 0.1))

    fig, axes = plt.subplots(nrows=1, ncols=len(clfs), figsize=(14, 4))

    for ax, (clf_name, clf) in zip(axes, clfs.items()):
        Z = clf.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]
        Z = Z.reshape(xx.shape)
        ax.contourf(xx, yy, Z, alpha=0.4)
        ax.scatter(X[:, 0], X[:, 1], c=y, s=20, edgecolor='black')
        ax.set_title(clf_name)

Below we plot a toy dataset which is not linearly separable. In the first example below we use a large number for the n_positive parameter in the make_imbalanced_blobs function (expand cell above). This effectively creates a fairly balanced dataset. We plot the decision boundaries for the two ensemble classifiers we used above.

X, y = make_imbalanced_blobs(n_positive=200)
plot_blobs_with_boundaries(get_clfs(sampling_strategy='auto'), X, y)

The bag of logistic regressions generates a near-linear decision boundary, so it is unable to effectively separate the classes. As expected, the balanced forest easily separates this dataset.

Now let's examine the imbalanced version of the dataset. The new decision boundaries are plotted below.

X, y = make_imbalanced_blobs(n_positive=20)
plot_blobs_with_boundaries(get_clfs(sampling_strategy=0.5), X, y)

We see that the decision boudary for the bag of logistic models has changed substantially, and is now non-linear. It is also better able to fit the dataset. In comparison, the balanced forest decision boundary is mostly the same.

Can we use bagging with a low-variance estimator like logistic regression?

As we see above, when we apply the bag of logistic regressions to a balanced dataset, most of the logistic models will tend to agree on the fitted coefficients, resulting in a near-linear boundary. There is little benefit of combining many nearly-identical models.

On the other hand, when using imbalanced datasets, the individual models exhibit high variance in the fitted weights, so combining them in a bag might produce a robust classifier.

Conclusion

Ensemble models provide a natural ways of dealing with imbalanced datasets, by incorporating a re-balancing procedure in the resampling. Logistic regression is not commonly used as a base model in bagging since it is a low-variance estimator. In imbalanced settings, even logistic regression can exhibit high variance. This is especially the case as we increase the dimensionality of the problem. In those cases, a bag of logistic regressions might produce competitive results.

Recently I moved to an apartment building. While waiting for the elevator one morning, I started thinking about writing a simplified event-based elevator simulation. Below I describe my attempt.

First, definitions and concepts

Why would we want to use generators to model elevators? From the Python docs:

Generator:A function which contains yield expression. Each yield temporarily suspends processing, remembering the location execution state (including local variables and pending try-statements). When the generator iterator resumes, it picks up where it left off You can see that the feature of "picks up where it left off" can be particularly useful when modeling elevators (and other event-based systems).

Concept inventory

elevator
elevator group
group of people - RENAME!

from collections import namedtuple
from queue import PriorityQueue

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# passenger priority is given by arrival time
Passenger = namedtuple('Passenger', ['arrival_time', 'id', 'start_level', 'end_level'])
Journey = namedtuple('Journey', ['elev_id', 'passenger_id', 'time_pickup', 'travel_time'])
ElevReady = namedtuple('ElevReady', ['ready_time', 'elev_id', 'level'])

FIXED_TIME = 15 # [s] time for each pickup
SAME_FLOOR_TIME = 5  # [s]
LEVEL_TIME = 0.5 # [s / level]
TOP_LEVEL = 40

def elev_time(start, end):
    if start == end:
        return SAME_FLOOR_TIME
    return abs(end - start) * LEVEL_TIME + FIXED_TIME

def elevator(elev_id, level=1):
    journey = None
    while True:
        group = yield journey
        time_pickup = elev_time(level, group.start_level)
        travel_time = elev_time(group.start_level, group.end_level)
        journey = Journey(elev_id, group.id, time_pickup, travel_time)
        level = group.end_level  # elevator waits at destination

class ElevGroup:
    """A group of elevators."""
    def __init__(self, n_elevs, elevator):
        assert n_elevs >= 1, 'Need at least one elevator.'
        self.elevs = {i: elevator(i) for i in range(n_elevs)}
        # at t = 0 all elevators are ready at floor = 1
        self.elevs_ready = [ElevReady(0, i, 1) for i in range(n_elevs)]
        for elev in self.elevs.values(): next(elev)  # prime generators
            
    def ready_for_passenger(self, passenger):
        return [r for r in self.elevs_ready if passenger.arrival_time >= r.ready_time]
     
    def dispatch_elev(self, passenger):
        """Get the next elevator for a given passenger.
        If more than one elevators are available, use the one closest to the passenger.
        
        Returns:
        The ElevReady event and corresponding elevator.
        """
        ready_elevs = self.ready_for_passenger(passenger)
        if ready_elevs:
            # pick the one closest to passenger.start_level
            def level_dist(elev_ready):
                return abs(elev_ready.level - passenger.start_level)
            ready_event = min(ready_elevs, key=level_dist)
        else:
            ready_event = min(self.elevs_ready, key=lambda r: r.ready_time)
        
        self.elevs_ready.remove(ready_event)
        return ready_event, self.elevs[ready_event.elev_id]

    def move(self, passenger):
        """Move a passenger using an elevator.
        Return the passenger wait time."""
        ready_event, elev = self.dispatch_elev(passenger)
        journey = elev.send(passenger)
        elev_depart_time = self.get_elev_depart_time(passenger, ready_event)
        next_ready_time = elev_depart_time + journey.time_pickup + journey.travel_time
        self.elevs_ready.append(ElevReady(next_ready_time, ready_event.elev_id, passenger.end_level))
        return self.get_wait_time(elev_depart_time, journey, passenger)
    
    @staticmethod
    def get_elev_depart_time(passenger, ready_event):
        return max(ready_event.ready_time, passenger.arrival_time)
    
    @staticmethod
    def get_wait_time(elev_depart_time, journey, passenger):
        return elev_depart_time + journey.time_pickup - passenger.arrival_time

def passengers_leave_building(n_passengers, interarrival):
    passenger_ids  = np.arange(n_passengers)
    start_levels   = np.random.randint(2, TOP_LEVEL + 1, size=n_passengers)
    end_levels     = np.ones(n_passengers)
    arrival_times  = np.cumsum(np.random.exponential(scale=interarrival, size=100))
    passenger_data = zip(arrival_times, passenger_ids, start_levels, end_levels)
    return [Passenger(*item) for item in passenger_data]

N_SIMUL = 200
N_PASSENGERS = 100

def simulate(n_elevs, passengers):
    elev_group = ElevGroup(n_elevs, elevator)
    passenger_queue = PriorityQueue()
    for p in passengers: passenger_queue.put(p)
    wait_times = []
    while not passenger_queue.empty():
        wait_times.append(elev_group.move(passenger_queue.get()))
    return np.array(wait_times)

def simul_many_passengers_leave(n_elevs, n_passengers, interarrival, seed=42):
    np.random.seed(seed)
    while True:
        passengers = passengers_leave_building(n_passengers, interarrival)
        yield simulate(n_elevs, passengers)
        
def collect_stats(simul, n_simul, stat_funcs):
    stats = np.zeros((n_simul, len(stat_funcs)))
    for s in range(n_simul):
        wait_times = next(simul)
        stats[s, :] = [f(wait_times) for f in stat_funcs]
    return stats
        
def plot_wait_times(wait_times, ax=None):
    ax = ax or plt.gca()
    ax.bar(np.arange(len(wait_times)), wait_times)
    ax.set_ylabel('Wait time [s]', fontsize=12)
    return ax

def plot_stats(stats):
    fix, axes = plt.subplots(ncols=2, figsize=(12, 3.5))
    axes[0].hist(stats[:, 0], bins=20)
    axes[0].set_title('Max wait time')
    axes[1].hist(stats[:, 1], bins=20)
    axes[1].set_title('Proportion waiting more 2 min')
    return axes

def prop_exceeds(wt, threshold=120):
    return (wt > threshold).mean()

s_3_20 = simul_many_passengers_leave(n_elevs=3, n_passengers=N_PASSENGERS, interarrival=20)

fix, axes = plt.subplots(ncols=2, figsize=(12, 3.5))
plot_wait_times(next(s_3_20), axes[0]);
plot_wait_times(next(s_3_20), axes[1]);

stats_3_20 = collect_stats(s_3_20, N_SIMUL, [np.max, prop_exceeds])
plot_stats(stats_3_20);

s_2_20 = simul_many_passengers_leave(n_elevs=2, n_passengers=N_PASSENGERS, interarrival=20)

fix, axes = plt.subplots(ncols=2, figsize=(12, 3.5))
plot_wait_times(next(s_2_20), axes[0]);
plot_wait_times(next(s_2_20), axes[1]);

stats_2_20 = collect_stats(s_2_20, N_SIMUL, [np.max, prop_exceeds])
plot_stats(stats_2_20);

stats = {}
for interarr in range(10, 70, 10):
    simul = simul_many_passengers_leave(n_elevs=2, n_passengers=N_PASSENGERS, interarrival=interarr)
    stats[interarr] = collect_stats(simul, N_SIMUL, [np.max, prop_exceeds])

sns.boxplot([d[:, 1] for d in stats.values()], x=range(10, 70, 10));

/home/nikolay/.pyenv/versions/3.7.4/envs/main/lib/python3.7/site-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-61-1eda1a77f112> in <module>
----> 1 sns.boxplot([d[:, 1] for d in stats.values()], x=range(10, 70, 10));

~/.pyenv/versions/3.7.4/envs/main/lib/python3.7/site-packages/seaborn/_decorators.py in inner_f(*args, **kwargs)
     44             )
     45         kwargs.update({k: arg for k, arg in zip(sig.parameters, args)})
---> 46         return f(**kwargs)
     47     return inner_f
     48 

~/.pyenv/versions/3.7.4/envs/main/lib/python3.7/site-packages/seaborn/categorical.py in boxplot(x, y, hue, data, order, hue_order, orient, color, palette, saturation, width, dodge, fliersize, linewidth, whis, ax, **kwargs)
   2230     plotter = _BoxPlotter(x, y, hue, data, order, hue_order,
   2231                           orient, color, palette, saturation,
-> 2232                           width, dodge, fliersize, linewidth)
   2233 
   2234     if ax is None:

~/.pyenv/versions/3.7.4/envs/main/lib/python3.7/site-packages/seaborn/categorical.py in __init__(self, x, y, hue, data, order, hue_order, orient, color, palette, saturation, width, dodge, fliersize, linewidth)
    404                  width, dodge, fliersize, linewidth):
    405 
--> 406         self.establish_variables(x, y, hue, data, orient, order, hue_order)
    407         self.establish_colors(color, palette, saturation)
    408 

~/.pyenv/versions/3.7.4/envs/main/lib/python3.7/site-packages/seaborn/categorical.py in establish_variables(self, x, y, hue, data, orient, order, hue_order, units)
    155             # Figure out the plotting orientation
    156             orient = infer_orient(
--> 157                 x, y, orient, require_numeric=self.require_numeric
    158             )
    159 

~/.pyenv/versions/3.7.4/envs/main/lib/python3.7/site-packages/seaborn/_core.py in infer_orient(x, y, orient, require_numeric)
   1303             warnings.warn(single_var_warning.format("Vertical", "x"))
   1304         if require_numeric and x_type != "numeric":
-> 1305             raise TypeError(nonnumeric_dv_error.format("Horizontal", "x"))
   1306         return "h"
   1307 

TypeError: Horizontal orientation requires numeric `x` variable.

Clustering is a messy business. It is hard to find literature on best practices in a business context. How to choose a clustering model, and a proper distance measure for a given dataset? How to select the optimal number of clusters? In published research, scientist often apply clustering on datasets originally used for classification, and measure clustering performance against the ground truth labels. This gives us some insight about the applicability of the clustering method on the type of data used in the study, but it does little to show how clustering can be applied in a business setting, where there are typically no labels, and we need to rely heavily on domain knowledge.

In practice, clustering is most useful when we don't have labels and we are not even sure what the labels might be.

Clustering on Survey Data

Well-designed surveys can help us better understand our customers. Survey data can be particularly valuable if coupled with additional metrics on how the survey respondents use the product in questions.

In this tutorial we apply Spectral Coclustering on the 2019 Kaggle Machine Learning and Data Science Survey. At the end of each year Kaggle sends out this survey to their (very large) user base, with the aim of capturing a snapshot of the state of ML and Data Science. We don't have access to data on how the survey respondents use Kaggle, so instead we focus on the survey data itself. We will use coclustering to find segments of data science practitioners, and interpret the factors unique to each segment.

Along the way we discuss a few technical topics in greater detail. In the next section, we cover the adjusted Rand index as a way of measuring the agreement between two data partitions (e.g. cluster solutions). In the last section, we discuss how Uniform Manifold Approximation and Projection (UMAP) can be applied to the survey dataset. Feel free to skip these sections if you are already familiar with these techniques.

Why Spectral Coclustering for survey data?

Spectral Coclustering clusters both rows and columns of a dataset, such that each row belongs to one cluster, and similarly, each column belongs to a single cluster. If we sort both the rows and columns based on the cluster labels, we obtain a block-diagonal structure. This structure can be clearly seen in this example where spectral coclustering was applied on a synthetic dataset. In part 2 of this tutorial (soon to come), we will dive deeper into the inner workings of the algorithm.

Clustering both rows and columns is particularly useful with medium-to-large surveys that have a lot of multiple-choice questions, like the Kaggle survey. Below, we select 23 questions for the evaluation, which results in 231 unique responses (columns in the data table). Using standard clustering it is difficult to compare the different user-clusters across so many different columns. With Spectral Clustering, we can focus on the responses that are most pertinent to a given cluster. Let's take a look at the toy example from the picture above. The corresponding table with responses is shown below.

#collapse
import json
from collections import defaultdict

import altair as alt
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import pandas as pd
from tqdm import tqdm
from umap import UMAP
import umap.plot

from sklearn.cluster import SpectralCoclustering
from sklearn.metrics.cluster import adjusted_mutual_info_score, adjusted_rand_score

alt.data_transformers.disable_max_rows();

Here each column corresponds to a response to a multiple-choice questions from a survey, so the responses are encoded as binary features. For example, the first two columns represent responses to the question: "Select an activity that make an important part of your role at work." If we run biclustering on this dataset, we might obtain two clusters, color-coded orange and blue in the picture above. The blue cluster would contain rows (users) that tend to mark responses such as "build prototypes" (important activity at work), "Tensorflow" (what framework do you use?), and "CNNs" (what machine learning models do you use?). In this way, we segment the users and characterize them at the same time. Similarly, the orange cluster would be described by "analyze and understand data", "SQL" and "ggplot2". Of course, the separation is not perfect, and you can see "blue" users occasionally selecting "orange" responses and vice versa.

The Kaggle Survey questions

One of the most important decision when clustering survey responses is choosing the questions used for clustering, and the ones used for analysis and validation post-clustering. If we were doing this in a professional setting, this would be a good time to discuss the survey goals with our teammates.

Most of the questions in this survey ask about things data science practitioners do at work: the types of problems they solve, frameworks they use, models they build. So we will use the responses to these questions for clustering. The full list of questions, color-coded based on my selection, is available here.

The questions selected for analysis (post-clustering) relate to the users' prior experience (coding experience, formal education), as well as team size. This selection will allow us to make conclusions such as: Users from cluster X focus on building deep learning prototypes using Tensorflow. Most of them have at least 2 years of experience with coding for data science.

Question 5 ("select title most similar to your current role") is used to select an initial number of clusters - more on that below.

The respondents

Once we drop empty or nearly-empty responses, we are left with records from more than 18000 users. However, the survey was designed such that users who don't have lot of programming experience for data analysis (Q15) were not shown many of the questions. This effectively creates two groups: one that did not see most of the questions, and one that did. For the rest of the tutorial we focus on the more-experienced group since it likely better represents data science practitioners (and they also filled out most of the questions, so we have more interesting data to work with). If you are curious about the preprocessing steps, you can find them in this notebook.

#collapse
df = pd.read_feather('data/encoded_subset.feather')
res = pd.read_feather('data/processed_subset.feather')

with open('data/responses.json', 'r') as f:
    uniques = json.load(f)
    
with open('data/short_questions.json', 'r') as f:
    short_qs = json.load(f)
    
qs = pd.read_csv('data/questions_only.csv').T.squeeze()

The questions used in clustering include single- or multi-selections, so each selection (column) can be encoded as binary.

df.head()

We also include mappings between question numbers and short summaries of the question, as well as the actual responses. These will help us interpret the clusters later.

short_qs['Q20'], uniques['Q20_Part_1'], uniques['Q20_Part_2']

('viz libraries', 'Ggplot / ggplot2', 'Matplotlib')

Selecting the number of clusters

There are a few important considerations when selecting the number of clusters for this type of task:

Most humans will struggle to keep track of more than about 10 user segments. Since the overall purpose of this analysis is to better understand our users, we will need to communicate our results to business leaders: we can anticipate that slides will be drawn, presentations will be given, and reports will be written. Limiting ourselves to a few segments will help us draw a clearer picture of the user-base, especially when getting started.
When doing the analysis, it is easy to overcluster (pick more clusters than what we expect the right number is), and then manually merge similar clusters together. Going in the opposite direction is harder.

Keeping these issues in mind, we will vary the number of clusters and measure the agreement between the resultant cluster solution and the responses from Question 5 (job role). We use this question because we would expect different data science segments to have distinct distributions of job roles. Furthermore, this question involves a single selection, so we obtain a single label (job role) per user. Note that Q5 is not our gold standard in terms of a cluster solution, we merely use it as a rough guide.

In terms of the actual metric to measure the agreement between the job role and cluster label, we can use the adjusted Rand index (ARI). It is a useful measure as it is corrected against chance - we delve into the details in the next section.

What about cluster evaluation metrics that do not require ground truth labels? Why not use silhouette coefficient or the Calinski-Harabasz index? Personally, I have not found these particularly useful, especially since each one tends to favor a particular type of clustering. If you have successfully used any of these metrics, let me know in the comments!

Rand Index

The Adjusted Rand Index is based on the Rand Index, so we need to understand the latter first. The Rand Index is a standard measure of the similarity of two partitions (e.g. cluster solutions) of a dataset. Let's suppose we want to measure the agreement between the following partitions:

from itertools import combinations

p1 = [0, 0, 0, 1, 1, 1, 2, 2, 2]
p2 = [0, 1, 0, 1, 1, 1, 2, 0, 2]

The Rand Index is calculated by looking at all pairs of items and finding:

a = number of pairs that are in the same subset in p1 and also in the same subset in p2.
b = number of pairs that are in different subset in p1 and also in different subset in p2.

The following functions do this calculation.

def same_subset(items, part):
    """Given a pair of items, find out if they are in the same subset"""
    i, j = items
    return part[i] == part[j]

def set_pairs(part, same=True):
    """Given a partition, find the set of pairs that are in the 
    same (same=True) or different (same=False) partitions"""
    if same:
        selection = lambda pair: same_subset(pair, part)
    else:
        selection = lambda pair: not same_subset(pair, part)
    n = len(part)
    return set(filter(selection, combinations(range(n), 2)))

For example, these are the set of pairs that are in the same subsets in p1:

set_pairs(p1, same=True)

{(0, 1), (0, 2), (1, 2), (3, 4), (3, 5), (4, 5), (6, 7), (6, 8), (7, 8)}

Once we can find the appropriate pairs, we can easily calculate the Rand Index. It is simply given by:

$$RI = \frac{a + b}{n_{pairs}} = \frac{2 (a + b)}{n (n - 1)}$$

where $n$ is the total number of items in our dataset.

def rand_index(p1, p2):
    """Compute the Rand index (not adjusted)"""
    assert len(p1) == len(p2) and len(p1) > 2
    a = len(set_pairs(p1, True) & set_pairs(p2, True))
    b = len(set_pairs(p1, False) & set_pairs(p2, False))
    n_pairs = len(p1) * (len(p1) - 1) / 2
    return (a + b) / n_pairs

rand_index(p1, p2)

0.75

So this metric varies between 0 and 1, where 1 indicates a perfect agreement between the two partitions. An important property is that the Rand index is independent of the actual cluster labels:

p3 = [7, 7, 7, 8, 8, 8, 9, 9, 9]
rand_index(p3, p2)

0.75

Adjusted Rand Index

One of the issues with the $RI$ is that it is not adjusted for chance. For example, we can take a look at random partitions of the same length as p1 and p2, and with the same number of subsets (3), and measure the $RI$ between these.

def rand_sample(u1, u2, part_len, n_iter):
    """Measures the Rand index between random sequences n_iter times.
    The two sequences have u1 and u2 unique elements, and are both of len = part_len."""
    scores = np.zeros(n_iter)
    for i in range(n_iter):
        s1 = np.random.choice(u1, part_len)
        s2 = np.random.choice(u2, part_len)
        scores[i] = rand_index(s1, s2)
    return scores

scores = rand_sample(3, 3, part_len=9, n_iter=1000)
plt.hist(scores, bins=10);
plt.title('Rand scores (random sequences len = 9)');

We see that $RI$ values between 0.4 and 0.7 are quite common when dealing with such a short sequence. We can try the same with longer sequences, and this is what we obtain:

scores_long = rand_sample(3, 3, part_len=100, n_iter=1000)
plt.hist(scores_long, bins=10)
plt.title('Rand scores (random sequences len = 100)');

The score for the longer sequences is quite sharply peaked between 0.54 and 0.58, so it will be very unlikely to obtain a value of 0.75 by chance.

The adjusted Rand index then takes this random variation into account as follows:

$$ARI = \frac{RI - E_{random}[RI]}{Max[RI] - E_{random}[RI]}$$

$ARI$ is simply a rescaled version of $RI$, computed by subtracting the mean $RI$ we would obtain by chance with random sequences (e.g. $E_{random}[RI] = 0.55$ for the first example above), and by dividing this value by the difference between the maximum and the average $RI$. The division is done to increase the sensitivity of the metric. This means that $ARI$ might be negative, if $RI < E_{random}[RI]$.

$ARI$ can be calculated exactly using the contingency table calculated from the two partitions. The formula can be found here, and it is used in the scikit-learn implementation.

However, I find it easier to understand the $ARI$ correction using the rand_sample function we saw above. This will give us an approximation, and it will be slower to compute since we need to sample many times to simulate the $RI$ distribution. On the other hand, we can clearly see the link between the code below and the $ARI$ definition:

def adjusted_rand(p1, p2, n_iter=5000):
    u1, u2 = len(np.unique(p1)), len(np.unique(p2))
    mean_rand = np.mean(rand_sample(u1, u2, len(p1), n_iter))
    ri = rand_index(p1, p2)
    return (ri - mean_rand) / (1 - mean_rand)

In the above code, I have used the simplification $Max[RI] = 1$. This would not work if the two sequences have different number of clusters.

adjusted_rand(p1, p2)

0.4362527091189256

Note that the $ARI$ is substantially lower for our short sequences compared with the $RI = 0.75$. We can also calculate it on longer sequences and compare against the scikit-learn implementation.

p4, p5 = np.random.choice(3, 100), np.random.choice(3, 100)
p4[:50] = p5[:50]  # some overlap between the sequences

adjusted_rand(p4, p5, 1000)

0.29431766785158325

from sklearn.metrics import adjusted_rand_score
adjusted_rand_score(p4, p5)

0.2925695663377777

The values are off by about 0.005, quite close!

Survey Analysis

We are now ready to do some clustering! We will fit SpectralCoclustering model using a range of clusters, and keep track of ARI.

#collapse
def cocluster_iter(range_cl, data, **kwargs):
    """Generator that fits coclustering for a range of different clusters."""
    for n_clusters in tqdm(range_cl):
        bicl = SpectralCoclustering(n_clusters=n_clusters, **kwargs)
        bicl.fit(data)
        yield n_clusters, bicl
        
def cluster_metrics(cocluster, metric, target):
    """Compute a given metric for a range of different n_clusters
    
    cocluster: coclustering generator that yields a fitted cluster object
    metric: metric callable, needs to accept metric(reference_labels, predicted_labels)
    target: target variable against which to compute metrics.
    """
    metrics = defaultdict(list)
    for n_clusters, bicl in cocluster:
        metrics['metric'].append(metric(target, bicl.row_labels_))
        metrics['n_clusters'].append(n_clusters)
    return pd.Series(metrics['metric'], index=metrics['n_clusters'])

cocluster = cocluster_iter(range(2, 25), df, random_state=0)
metrics = cluster_metrics(cocluster, adjusted_rand_score, res['Q5'].cat.codes)
ax = metrics.plot()
ax.set_title('Cluster metric against Q5 (Occupation)');

100%|██████████| 23/23 [00:58<00:00,  2.56s/it]

There is a sudden spike at n_clusters=9, so we will use that as a starting point.

%%time

bicl = SpectralCoclustering(n_clusters=9, random_state=0)
bicl.fit(df)

print('Column (question-response) cluster counts:', np.unique(bicl.column_labels_, return_counts=True)[1])
print('Row (user) cluster counts:', np.unique(bicl.row_labels_, return_counts=True)[1])

Column (question-response) cluster counts: [31 30 11  9 11  6 35 28 70]
Row (user) cluster counts: [1914 2611 1113  856  565  123 2703 2330 1471]
CPU times: user 12.8 s, sys: 2.95 s, total: 15.7 s
Wall time: 2.06 s

We obtain relatively balanced clusters, where the smallest user cluster contains 123 users, while the largest contains 2703 users. No single cluster dominates the questions or the users.

Clusters and job roles

First, we study the relationship between Q5 and the clusters. We use altair for plotting, because it allows us to easily add interactive elements such as tooltips to our plots. Here is a bar chart of the distribution of job roles per cluster. Since the actual cluster numbers are arbitrary, we sort the bars based on the proportion of students, since clusters with a large proportion of students should behave differently than those with few or no students. You can hover over the individual bars to get the actual user counts.

#collapse

# create an analysis table where some of the questions are ordered
row_labels = pd.Series(bicl.row_labels_, index=df.index, name='cluster')
analysis = pd.concat([res[['Q4', 'Q5', 'Q6', 'Q7', 'Q15']], row_labels], axis=1)

q15_map = {
    '< 1 years': '<= 2 years',
    '1-2 years': '<= 2 years',
    '3-5 years': '3-10 years', 
    '5-10 years': '3-10 years',
    '10-20 years': '10+ years',
    '20+ years': '10+ years',
}
simple_cats = ['<= 2 years', '3-10 years', '10+ years']
analysis['Q15_simple'] = pd.Categorical(analysis['Q15'].map(q15_map), categories=simple_cats,
                                        ordered=True)
for name, col in analysis.iteritems():
    if hasattr(col, 'cat') and col.cat.ordered:
        analysis[f'{name}_order'] = col.cat.codes + 1

#collapse

# sort clusters by proportion of students in the cluster
sort_order = (analysis
              .groupby('cluster')['Q5']
              .apply(lambda s: (s == 'Student').mean())
              .sort_values())

alt.Chart(data=analysis[['Q5', 'cluster']]).mark_bar(size=35).encode(
    x=alt.X('cluster:N', sort=sort_order.index.tolist()),
    y=alt.Y('count()', stack='normalize', title='Proportion (per cluster)'),
    color=alt.Color('Q5', scale=alt.Scale(scheme='tableau20')),
    tooltip=['Q5', 'count()']
).properties(
    width=500
)

We can see that clusters on the left are dominated by Data Scientist, and followed by Software Engineer, Data Analyst and Research Engineer. Besides Student, clusters on the right have higher Not employed proportion.

Average responses per cluster

To get any further into the analysis, we need to examine the relationship between the row and column clusters. To do this, we need to do a bit of data wrangling. First, we are going to sort our data matrix according to the cluster labels.

#collapse
row_idx, col_idx = np.argsort(bicl.row_labels_), np.argsort(bicl.column_labels_)
df_sorted = df.iloc[row_idx, col_idx]
rows_sorted = bicl.row_labels_[row_idx]
cols_sorted = bicl.column_labels_[col_idx]

Since some responses are inherently more popular than others, we are first going to subtract the average per column (computed across all users).

We can visualize the data matrix, after the subtraction, as follows:

#collapse
df_center = df_sorted - df_sorted.mean()

fig = plt.figure(figsize=(7, 5))
ax = plt.gca()
im = ax.imshow(df_center, aspect=0.015, cmap='PRGn')
cbar = fig.colorbar(im, ax=ax)
cbar.ax.set_ylabel('Subtracted value', rotation=-90, va="bottom")
fig.tight_layout()

This heatmap gives us a good global picture of the dataset. Both rows and columns are sorted based on cluster: for example, cluster 0 is at the top left. Most of the high values in green are along the block diagonal, as we would expect from the coclustering. However, there are a few green blocks away from the diagonal, in particular in the lower left corner. We will need to keep an eye on those especially when we analyze clusters 7 and 8, which are the ones with the largest off-diagonal blocks.

Next, we are going to display average values per cluster, so we can characterize each cluster. First, we are going to create better labels for our columns. Currently, each column looks like this: Q27_Part_4; it is time-consuming to look up the questions and responses. Instead, what we want is something more readable, such as: 'NLP methods: Transformer language models (GPT-2, BERT,' We have all the necessary information in the uniques and short_qs mappings.

#collapse
def col_label(col, n_words=5):
    """Make a human-friendly (but short) label for a given column.
    'Q26_Part_2' -> 'CV methods: Image segmentation methods (U-Net, Mask'
    """
    answer_words = uniques[col].split()
    answer_label = ' '.join(answer_words[:n_words])  # limit to n_words
    q_index = col.split('_')[0]  # 'Q26'
    q_label = short_qs[q_index]  # 'CV methods'
    return q_label + ': ' + answer_label

col_label('Q27_Part_3')

'NLP methods: Contextualized embeddings (ELMo, CoVe)'

We can now compute averages per cluster (using the centered data), and add the labels we created.

#collapse
avg_cluster = df_center.groupby(rows_sorted).mean()
avg_cluster.columns = avg_cluster.columns.map(col_label)
avg_cluster.head(2)

From the above, we note that cluster 0 is much more likely to use Image classification than cluster 1, for example. Using df_center helps emphasize the differences between the clusters.

We are finally ready to summarize the clusters. Based on the heatmap above, we will focus mostly on the cluster-specific responses (columns) that run along the block diagonal: for a cluster of users c, these are the columns that belong to column cluster c. We will also keep an eye for high values in the rest of the columns. Below is a simple function which will print a cluster summary, that is, the columns with the highest average (centered) score for a given cluster.

#collapse
def get_cluster_cols(c):
    """Split the average responses for a given cluster into two groups:
    specific: these are the responses (columns) specific to a given cluster
    others: the rest of the responses, that belong to a different cluster"""
    specific = avg_cluster.loc[c, cols_sorted == c].sort_values(ascending=False)
    others = avg_cluster.loc[c, cols_sorted != c].sort_values(ascending=False)
    return specific, others

def print_summary(c, n_specific=10, n_other=5):
    print('Cluster', c)
    specific, others = get_cluster_cols(c)
    print('Cluster-specific:')
    print(specific.head(n_specific).round(3).to_string())
    print()
    print('Others (off-diagonal):')
    print(others.head(n_other).round(3).to_string())
    print()

print_summary(0, n_specific=15)

Cluster 0
Cluster-specific:
ML algos regular: Convolutional Neural Networks                0.494
CV methods: Image classification and other general             0.385
hardware: GPUs                                                 0.343
ML frameworks: Keras                                           0.309
hosted notebooks: Google Colab                                 0.303
ML frameworks: TensorFlow                                      0.287
ML algos regular: Dense Neural Networks (MLPs, etc)            0.268
CV methods: Image segmentation methods (U-Net, Mask            0.263
CV methods: General purpose image/video tools (PIL,            0.263
ML algos regular: Recurrent Neural Networks                    0.247
ML frameworks: PyTorch                                         0.240
CV methods: Object detection methods (YOLOv3, RetinaNet,       0.225
language regular: C++                                          0.176
ML tools regular: Automated data augmentation (e.g. imgaug,    0.171
CV methods: Generative Networks (GAN, VAE, etc)                0.128

Others (off-diagonal):
hosted notebooks: Kaggle Notebooks (Kernels)                        0.188
viz libraries: Matplotlib                                           0.182
NLP methods: Word embeddings/vectors (GLoVe, fastText, word2vec)    0.156
language recommend: Python                                          0.121
IDEs regular: Jupyter (JupyterLab, Jupyter Notebooks, etc)          0.103

Based on this, users in cluster 0 are very focused on deep learning and computer vision. Let's contrast them with users from cluster 7 - this is one of the clusters whose responses are overlapping with those of cluster 0.

We see that cluster 7 is also focused on deep learning as well as experimentation and iteration and research. When we go back to our job role bar-chart, we note that cluster 0 contains the highest proportion of students across all clusters, whereas cluster 7 is a mix of mostly data scientists, software engineers and research scientists.

By keeping track of the off-diagonal elements, we are able to better characterize the two clusters. We can go a little further by examining differences in responses to Q15 ("years of writing code for data analysis")

# collapse
def stacked_bar_cluster(data, q):
    """Stacked bar with order information.
    
    data: dataframe to plot, needs to contain the question, and cluster
    q: name of the question, e.g. 'Q6'
    """
    q_order = f'{q}_order'
    order = q_order if q_order in data.columns else []
        
    return alt.Chart(data=data).mark_bar(size=25).encode(
        x=alt.X('cluster:N', sort=sort_order.index.tolist()),
        y=alt.Y('count()', title='Proportion (per cluster)', stack='normalize'),
        # need to provide a list with ordered categories to display correctly
        color=alt.Color(f'{q}:O', scale=alt.Scale(scheme='inferno'), 
                        sort=list(data[q].cat.categories)),
        tooltip=[q, 'count()'],
        # force an order on a categorical variable
        order=order
    ).properties(
        width=400,
        height=280,
    )

stacked_bar_cluster(analysis, 'Q15_simple').properties(title='Years writing code (data)')

The majority of cluster 0 users (65%) are fairly new to programming for data analysis (<= 2 years). In the case of cluster 7, this proportion is much smaller, at about 35%. In summary, there is some overlap in terms of the technologies that the two clusters are using but their experience and job roles are substantially different.

#collapse
stacked_bar_cluster(analysis, 'Q6').properties(title='Size of company')

Students were not asked about the size of the company, which is why we see the null values.

Data science segments

We can use this procedure (examine highest responses per cluster, cross-reference with the analysis questions) to characterize the rest of the clusters. I came up with four groups of clusters, and it took me about 30 minutes once I had the code above ready.

A note about the notation

Below I use a $\Delta$ when I quote the centered scores: for example, for cluster 0, the centered score for "using Keras" is $\Delta = 0.31$, and the actual average score is $0.76$. This means that 76% of users in cluster 0 have selected Keras, and this is a lot higher ($\Delta = 0.31$) when compared to the overall population.

#collapse
user_segments = {
    'learners': [0, 6],
    'beginners': [3, 4, 5],
    'R users': [2],
    'professionals': [1, 7, 8],
}
cluster_size = row_labels.value_counts(sort=False)
fig = plt.figure(figsize=(8, 5))
ax = fig.gca()
for i, (segment, clusters) in enumerate(user_segments.items()):
    ax.bar(clusters, cluster_size.loc[clusters], label=segment, 
           color=matplotlib.cm.Pastel1(i))
ax.legend()
ax.set_xticks(range(bicl.n_clusters));
ax.set_xlabel('clusters', fontsize=12)
ax.set_ylabel('number of respondents', fontsize=12);

Learners: Cluster 0, 6

These two clusters have the highest proportion of students (at about 50%). These are also the groups with least experience in terms of years of programming (<= 2 years for the majority of users in both clusters). Both groups are active users of Kaggle Notebooks and Courses.

Cluster 0 users tend to focus on deep learning, especially on computer vision tasks (image classification: $\Delta = 0.49$, image segmentation: $\Delta = 0.26$). They use Keras ($0.72$) and Tensorflow ($0.70$) for building models.

Respondents from Cluster 6 use more traditional Python data analysis tools, such as Matplotlib ($\Delta = 0.15$), Seaborn ($\Delta=0.15$) and Scikit-learn ($\Delta = 0.12$).

Beginners: Clusters 3, 4, 5

These are the smallest clusters and together account for about 11% of the respondents. These users respond to most of the questions with None. Excel (or other related products) is the main software tool that shows up in all three clusters, especially in cluster 5 with a score of $\Delta = 0.40$. The most common job roles are: Business/Data Analyst, Software Engineer, Student.

Depending on the business purposes, it might be reasonable to merge these clusters together in a single group. There are a few differences, however. For example, users from cluster 4 have longer coding experience (more than half of them have 3+ years of coding experience), and there are almost no students in that cluster.

R Users: Cluster 2

This is the R cluster! Examining the highest responses makes this clear: R as language regularly used ($\Delta = 0.58$), RStudio as the IDE ($\Delta = 0.57$), ggplot2 for visualization ($\Delta = 0.53$), and again, R, as the language to recommend ($\Delta = 0.27$). In terms of ML algorithms, these users are more likely to fit linear and logistic regression models ($\Delta = 0.16$).

Here, we see a split of the job role distribution between students, data scientists and data analysts. In terms of coding experience, this cluster is in-between the beginner clusters above and the professional clusters we will examine below. This cluster captures about 8% of the users.

Professionals: Clusters 1, 7, 8

These are the most experienced groups of users. Each of these clusters have unique characteristics, but we first review some of the similarities. These are users with the most coding experience (with more than 60% of users in each group having 3+ years of experience). There are no students in these clusters, and conversely, these are clusters with largest proportions of data scientists (almost half of users in cluster 1). Now, onto the unique aspects of each cluster:

Cluster 7 ML Researchers

We mentioned cluster 7 earlier when we contrasted it with cluster 0. Cluster 7 users focus on experimentation and iteration to improve ML models ($\Delta = 0.28$), and do research to advance the state of the art in ML ($\Delta = 0.18$). Curiously, these are also the users that are most likely to have an employer with a mature ML ecosystem ($\Delta = 0.16$). The research focus is deep learning, and captures almost all domains, libraries, and models of deep learning. On the other hand, these users are not likely to use cloud products (cloud products - None: $\Delta = 0.27$) and big data products (big data products - None: $\Delta = 0.45$). Perhaps, this is because these users are building custom models and products that are not well-supported by mainstream cluod services.

Cluster 1 Analysts and prototype builders

Cluster 1 users focus on analyzing data to influence product decisions ($\Delta = 0.28$) and building prototype to explore ML applications ($\Delta = 0.25$). There is some overlap with cluster 7, but it appears that cluster 1 users are focusing on tabular data: they are more likely to use SQL ($\Delta = 0.17$), as well as all major SQL databases. They use algorithms traditionally applied to tabular data such as decision trees and random forests ($\Delta = 0.20$), and gradient boosting ($\Delta = 0.16$). They are also more likely to use R regularly ($\Delta = 0.24$).

More than half of the users in cluster 1 work for a large company (1000+ employees), which is the highest proportion of any cluster.

Cluster 8 Deployment and Cloud

Users in cluster 8 focus on putting models to production using the cloud. Virtually all cloud-related platforms, services and products fall under this cluster. These users are also more likely to use use AutoML products (Auto-Sklearn: $\Delta = 0.25$ and Google AutoML: $\Delta = 0.24$). These users focus on building the necessary data infrastructure ($\Delta = 0.22$) as well as analyzing data to influence product decisions ($\Delta = 0.21$).

Interestingly, Google Cloud is more popular with these users ($0.64$) compared to Amazon Web Services ($0.57$), and Microsoft Azure ($0.38$), which is very different from the market share of each of these cloud providers (AWS is well ahead, followed by Microsoft and Google). The over-representation of Google Cloud in this survey might be due to (1) Google Cloud's strong offerings in the machine learning space and (2) the fact that Kaggle is part of Google and it promotes some Google products.

We should also point out that this is the group with the highest response rates across all clusters (note all the green values for the last row-block in the heatmap).

UMAP projection

We are going to conclude our analysis by creating a UMAP projection of our dataset on the 2D plane. UMAP is a very useful nonlinear dimensionality reduction technique and deserves a tutorial of its own. We are not going to delve into the details here, and simply use it with these goals in mind:

Assess the agreement between clustering labels and UMAP projection. If the two agree with each other, we can be more confidentent in our analysis.
UMAP projections, especially plots of the underlying connectivity matrix, often look (to use the technical term) pretty cool. Plots that are pretty cool often attract attenion: they can be included in title slides or document highlights and help get more people excited about out data analysis.

Jaccard coefficient as a distance measure

Perhaps the most important UMAP parameter is the distance measure. Our dataset includes binary features only (yes / no selections) so the Jaccard coefficient is a good choice. It is defined as the size of the intersection of two sets $u$ and $v$, divided by the size of their union:

$$J(u, v) = \frac{|u \cap v|}{|u \cup v |}$$

Wikipedia has a nice graphic of the Jaccard coefficient. In our case, the numerator will count the number of matches (shared selections) between users $u$ and $v$. The denominator will normalize this count by the total number of unique selections of both $u$ and $v$. It is easier to have more matches with a user that has made a lot of positive selections, so we need the denominator to control for this effect.

Let's compute the Jaccard index for a few of the users in the toy dataset to gain a better intuition. Here is the dataset again

The Jaccard index for users 0 and 1 is: $$J(u_0, u_1) = \frac{1}{6} = 0.167$$ because they have a single match (Analyze / understand data), and a total of 6 unique selections. On the other hand: $$J(u_0, u_2) = \frac{2}{4} = 0.50$$ Note that both the numerator and the denominator changed because the intersection of $u_0$ and $u_2$ is larger, while the union is smaller.

The Jaccard index appears in other places in machine leanring; for example it is often used for the evaluation of image segmentation models. It is also closely related to cosine similarity, which works with continuous features, in addition to binary label sets. Indeed, using metric='cosine' in the projection below results in a very similar projection.

#collapse
mapper = UMAP(n_neighbors=15, min_dist=0.1, 
              metric='jaccard', random_state=0).fit(df)

/home/nikolay/.pyenv/versions/3.7.4/envs/main/lib/python3.7/site-packages/umap/umap_.py:1530: UserWarning: gradient function is not yet implemented for jaccard distance metric; inverse_transform will be unavailable
  "inverse_transform will be unavailable".format(self.metric)

UMAP projection and cluster labels

First, we are going to plot the projection, colored by the cluster labels.

#collapse
umap.plot.points(mapper, labels=row_labels, color_key_cmap='tab10', 
                 width=700, height=600);

Overall, there is a good agreement between the cluster labels and the projection. Here are some of my observations:

There are two main structures in the projection (two wings of a butterfly?), with a clear separation between them. One includes mostly clusters with students (0, 2, 3, 5, 6) and the other one includes clusters without.
Learners (clusters 0 and 6) are neighboring each other in the projection, and so are the Professionals (1, 7 and 8).
The Beginners (3, 4, 5) are split into two groups, with clusters 3 and most of 5 on the one side.

UMAP connectivity matrix

We can also examine the connectivity matrix used by UMAP to create the projection.

#collapse
umap.plot.connectivity(mapper, edge_cmap='magma', background='black', 
                       width=700, height=600);

Indeed, the connectivity matrix looks cool! It will make a fine presentation highlight.

In addition to the dense local connections, there are many connections between the two large structures that run in parallel to each other. This provides some additional information about the structure of the data which is lost in the 2D scatter plot. These parallel connections are encouraging because it appears that some of them link cluster 4 with 3 and 5 (near the top of the structure) suggesting that there is indeed a similarity between these, as noted previously.

Conclusion

This was a long journey! So what did we discover along the way?

We found 6 segments of data science practitioners (after merging together the beginners group) based on 9 clusters. The segments are interpretable, and can potentially impact product offerings, especially if we could do additional analysis on how the different segments use the platform.
We learned that the Rand Index is an intuitive measure to calculate cluster agreement, but it needs to be adjusted for chance (ARI to the rescue!).
Finally, the UMAP projection (using the Jaccard index) showed some agreement with the clustering solution, and also produced a pretty cool connectivity plot.

Of course, we must always be cautious when we analyze survey results, and be aware of the different source of bias that might appear. This article lists major sources of bias in surveys.

I hope you found this tutorial useful! If you have any feedback, let me know in the comments below!

	Q9_Part_1	Q9_Part_3	Q9_Part_5	Q9_Part_6	Q12_Part_1	...	Q19_8	Q22_0	Q22_1	Q22_3
0	0	0	0	0	1	...	1	1	0	0
1	0	0	0	0	0	...	1	0	1	0
2	0	0	0	0	0	...	1	1	0	0
3	0	1	0	1	0	...	0	1	0	0
4	1	0	1	1	0	...	1	0	0	1

	used TPU: > 25 times	NLP methods: Encoder-decorder models (seq2seq, vanilla transformers)	CV methods: Image classification and other general	CV methods: Object detection methods (YOLOv3, RetinaNet,	CV methods: Image segmentation methods (U-Net, Mask	CV methods: General purpose image/video tools (PIL,	ML tools regular: Automated data augmentation (e.g. imgaug,	IDEs regular: PyCharm	ML algos regular: Recurrent Neural Networks	ML algos regular: Generative Adversarial Networks	...	Cloud platforms: VMware Cloud	ML products: Amazon SageMaker	ML products: Google Cloud Translation	ML products: RapidMiner	ML products: Google Cloud Natural Language	ML products: Google Cloud Speech-to-Text	ML products: Google Cloud Vision	ML products: Google Cloud Machine Learning Engine	Cloud platforms: SAP Cloud	Cloud products: AWS Elastic Compute Cloud (EC2)
0	0.012916	0.125789	0.384853	0.225243	0.263230	0.262646	0.17058	0.082068	0.246883	0.116438	...	-0.010134	-0.039215	-0.013583	-0.015739	-0.026209	-0.021901	-0.022826	-0.033771	-0.005115	-0.121536
1	-0.009063	-0.082646	-0.185662	-0.119416	-0.125876	-0.136143	-0.09547	-0.055362	-0.140269	-0.059602	...	-0.001221	0.001486	-0.015535	0.010130	-0.021892	-0.020930	-0.021469	-0.032381	0.001013	0.070342

	name	n_examples	n_feats	prop_positive	n_positive
0	ozone_level	2536	72	0.028785	73
1	coil_2000	9822	85	0.059662	586
2	abalone_16+	4177	10	0.062485	261

	Q9_Part_1	Q9_Part_3	Q9_Part_5	Q9_Part_6	Q12_Part_1	...	Q19_8	Q22_0	Q22_1	Q22_3
0	0	0	0	0	1	...	1	1	0	0
1	0	0	0	0	0	...	1	0	1	0
2	0	0	0	0	0	...	1	1	0	0
3	0	1	0	1	0	...	0	1	0	0
4	1	0	1	1	0	...	1	0	0	1

	Q9_Part_1	Q9_Part_3	Q9_Part_5	Q9_Part_6	Q12_Part_1	...	Q19_8	Q22_0	Q22_1	Q22_3
0	0	0	0	0	1	...	1	1	0	0
1	0	0	0	0	0	...	1	0	1	0
2	0	0	0	0	0	...	1	1	0	0
3	0	1	0	1	0	...	0	1	0	0
4	1	0	1	1	0	...	1	0	0	1

Nikolay’s blog

Discrete-event Coffee Shops

Discrete-event simulation

Simpy

Coffee Shop

Barristas and resources

Refactoring, refactoring

Have a little patience

Adding a cashier

Further exploration

Conclusion

Imbalanced Data and Bags of Logistic Regressions

Introduction

Datasets

Ozone level

Coil 2000

Abalone

Experiments

Models

Evaluation

Results

Bag of logistic regressions

Isn't a bag of linear models simply a linear model?

Can we use bagging with a low-variance estimator like logistic regression?

Conclusion

Elevators and Generators

First, definitions and concepts

Concept inventory

ML Engineering Workflow Tools

ML Engineering Workflow Tools

Why is Data Science code so messy?

Quick Overview of the Tools

DVC

MLFlow

Kedro

Conclusion

References

Spectral Coclustering to segment data science practitioners

Clustering on Survey Data

Why Spectral Coclustering for survey data?

The Kaggle Survey questions

The respondents

Selecting the number of clusters

Rand Index

Adjusted Rand Index

Survey Analysis

Clusters and job roles

Average responses per cluster

Data science segments

Learners: Cluster 0, 6

Beginners: Clusters 3, 4, 5

R Users: Cluster 2

Professionals: Clusters 1, 7, 8

UMAP projection

Jaccard coefficient as a distance measure

UMAP projection and cluster labels

UMAP connectivity matrix

Conclusion

	Q9_Part_1	Q9_Part_3	Q9_Part_5	Q9_Part_6	Q12_Part_1	...	Q19_8	Q22_0	Q22_1	Q22_3
0	0	0	0	0	1	...	1	1	0	0
1	0	0	0	0	0	...	1	0	1	0
2	0	0	0	0	0	...	1	1	0	0
3	0	1	0	1	0	...	0	1	0	0
4	1	0	1	1	0	...	1	0	0	1