
I've been spending lots of time looking at names [recently](/2022-05-22-machine-learning-name-interpolation). The US Government posts a [dataset](https://catalog.data.gov/dataset/baby-names-from-social-security-card-applications-national-data) of the yearly popularity of baby names from 1880. I wanted a nicer way to let the names tell me about the relationships between them. I therefore made a 3d name graph to let name historical data speak for itself and show how name popularity evolved over time.

![Name graphs for the top 15 girl names.](/images/names_per_year.png)

To keep it sane, I filtered by the allowed Danish names. I also cut off by popularity at some point, leaving in the ballpark of 1000 names per gender.

![Location of DK approved names in ordered list of all US names](/images/dknames-location.png)

I then created a graph where the nodes are names and the distances between them is the similarity between the two names' histories. For this I used mean squared error since I wanted to capture the generation difference, which might get lost with something like dynamic time warp.

To visualize, I made and deployed a simple react web app using a force directed graph visualization library. It takes in the name graph and and allows the nodes to arrange themselves neatly based on the link distances. I limit the number of links for performance reasons.

The app runs on [http://baby.gravi.la/](http://baby.gravi.la/). I recommend a large and wide screen with WebGL acceleration, since I didn't optimize it for mobile use. I provide datasets for boy and girl names in a few variants selected for better and worse performance.

#### Interpretation

Though completely speculative and knowing well there are many exceptions, I find it very interesting to see how the top 545 girl dataset arranges itself into a semicircle. The fact that it looks like it's working on completing itself validates the presence of some kind of cyclicality, usually named the 100 year rule. Names towards the center also tend to be names that have been popular longer and therefore tied to more periods. Otherwise it's clear to see the end containing the names popular 100 years ago, the names which are very trendy now, and everything in-between.


Making a name graph from their popularity over time


It's hard to decide on a baby name, even more so when multiple people are invested in the decision-making process. My friends recently had a baby and couldn't decide between Alex and Niklas. I naturally took it upon myself to solve their problem by creating a Machine Learning algorithm that finds the names laying perfectly in the middle. What do you get when you split the difference between Alex and Niklas?

## Approach

Variational AutoEncoders (VAEs) are a popular type of Machine Learning model that expands on autoencoders. I've presented autoencoders in some previous posts where I use them to [reduce dimensionality](/posts/2020-05-24-reduce-mnist-dimensionality-using-autoencoders/) or [perform clustering on unlabelled data](/posts/2020-05-31-clustering-on-encoding/). AEs are models that recreate their inputs, usually producing something interesting along the way. Since these intermediate products of the model need to contain all the information to recreate the original, they usually represent a deeper understanding of the data itself.

While AEs are good for compressing existing data, they are not suited for generating new data, since the space between existing embeddings is not necessarily meaningful within the latent space. VAEs solve this problem by using the encoder output to sample from a normal space, thereby regularising the latent output. In other words, a VAE is better at giving meaning to the embedding, therefore it makes more semantic sense to sample outside the existing data points. Since the desired outcome of this project is to be able to split the difference between the encodings of "Alex" and "Niklas", this architecture should provide the required functionality.

## Data

The data is available from [familieretshuset](https://familieretshuset.dk/navne/navne/godkendte-fornavne), which provides a list of over 20,000 boy names in Denmark. I converted the special characters å, æ and ø to a, ae, and o for simplicity. I also removed names exceeding 10 characters, since they represented an insignificant amount of the dataset.

## Architecture

The model I have settled on is a sequence to sequence (Seq2Seq) model using a combination of Conv1D and LSTM cells. The encoding is used as a repeated input to the decoder until it finishes producing the output. The latent vector has dimension 256 as the output of the encoder passes through two 256-neuron dense layers to produce the mean and variance vectors.

```raw
Model: "encoder"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 conv1d (Conv1D)             (None, None, 32)          2624

 conv1d_1 (Conv1D)           (None, None, 32)          3104

 conv1d_2 (Conv1D)           (None, None, 32)          3104

 bidirectional (Bidirectiona  (None, None, 128)        49664
 l)

 bidirectional_1 (Bidirectio  (None, None, 128)        98816
 nal)

 lstm_2 (LSTM)               (None, 64)                49408

=================================================================
Total params: 206,720
Trainable params: 206,720
Non-trainable params: 0
_________________________________________________________________

Model: "decoder"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 repeat_vector (RepeatVector  (None, 10, 256)          0
 )

 bidirectional_2 (Bidirectio  (None, 10, 128)          164352
 nal)

 bidirectional_3 (Bidirectio  (None, 10, 128)          98816
 nal)

 conv1d_3 (Conv1D)           (None, 10, 32)            12320

 conv1d_4 (Conv1D)           (None, 10, 32)            3104

 conv1d_5 (Conv1D)           (None, 10, 27)            2619

=================================================================
Total params: 281,211
Trainable params: 281,211
Non-trainable params: 0
_________________________________________________________________
```

I created a custom callback to compute the average edit distance between input and output names. The callback also stops the training when the model perfectly reconstructs over 50% of names. Since this is a toy project, I haven't spent more time perfecting the training loop. I also reduced the variance effect in order to more easily obtain results.

## Results

We can now use the machine learning model to interpolate between two names. First, we find where Alex and Niklas exist in the latent space.

```python
# Get encodings for Alex and Niklas
alex_encoded, _ = vae.encode(prepare("alex"))
niklas_encoded, _ = vae.encode(prepare("niklas"))
```

Just like getting points between a start and end point on a map in 2D space, we interpolate the points between Alex and Niklas in 256-dim space. We use these intermediate vectors as generation seeds for the decoder, producing all names between Alex and Niklas.

```python
# Get intermediate names from 100% Alex to 100% Niklas
for perc in np.arange(0,1.1,0.1):
    wsum = perc*niklas_encoded + (1-perc)*alex_encoded
    decoded = vae.decode(wsum)[0]
    decoded = ohe_to_name(decoded)
    print(f"{round((1-perc)*100):03d}% alex + {round((perc)*100):03d}% niklas -> {decoded}")
```

Finally, the output

```raw
100% alex + 000% niklas -> alex
090% alex + 010% niklas -> alex
080% alex + 020% niklas -> alex
070% alex + 030% niklas -> alex
060% alex + 040% niklas -> allx
050% alex + 050% niklas -> erli
040% alex + 060% niklas -> eillas
030% alex + 070% niklas -> nillas
020% alex + 080% niklas -> nillas
010% alex + 090% niklas -> niklas
000% alex + 100% niklas -> niklas
```

So there it is, halfway between Alex and Niklas is **Erli**, closely followed by Elias (close enough to eillas).


Using Machine Learning to Compromise between names


The quality of the datasets we use for training machine learning models is an often-discussed subject. I wanted to test the popular saying "garbage in, garbage out" by making a few toy scripts to explore the effect of dataset poisoning. For this, I use some popular datasets provided by Keras: MNIST, fashion-MNIST, CIFAR10 and CIFAR100. All four consist of images, and the model will perform classification.

The experiment shows that the model used is surprisingly robust even when it mislabels more than half of the dataset. Additionally, the same percentage of bad data has a greater effect on a smaller dataset.

[The code can be found on my Github.](https://github.com/felixgravila/dataset_poisoning)

## Poisoning the dataset

I train multiple models on varying proportions of bad data all the way from 0% to 100%. For the first round of experiments, I used a random wrong class as labels for the mislabeled data. Seeing the surprisingly high accuracies produced, I wondered what the difference would be if the errors were more systematic. The "biased" models mislabel data by always using the first available wrong class.

## Training

The focus of the experiment is on the obtained accuracy relative to data, so I have chosen a simple [sequential](/posts/2020-04-26-creating-a-custom-training-loop-in-tensorflow/) convolutional neural network as the model. Since the datasets have different difficulties, the model will perform worse on the harder ones.

I train a different set of models on a subset of the training datasets to investigate whether that changes anything. The intuition is that the effect of data poisoning is more pronounced on a smaller dataset.

The models train until their validation accuracy stops improving in over 10 epochs. Additionally, the learning rate reduces after 5 epochs of no improvement.

## Evaluation

I evaluated the models using the validation datasets provided, kept unmodified. Training on the full datasets shows the easy MNIST dataset maintaining a very high accuracy even with 80% of the data randomly mislabeled. However, the more complex datasets such as CIFAR-100 quickly dropped in accuracy even with small quantities of bad data. Biasing the wrong labels to almost always be a single class caused accuracy to start dropping quicker, but the models were still robust up to 40% poisoning compared to their random counterparts.

<figure>

![The effect of poisoning the full datasets](/images/poisoning_full_all.png)

<figcaption>

The effect of poisoning the full datasets.

</figcaption>

</figure>

I trained the models on the first 1,000 items from the training set, as this is a more reasonable quantity of labelled data to own compared to 60,000 (MNIST), 50,000 (CIFAR). In this case, the wrong labels affect the accuracy significantly earlier. The CIFAR100 model should be disregarded as it was unable to train with ~10 input images per class, even with no poisoning.

<figure>

![The effect of poisoning 1000 item datasets](/images/effect_of_poisoning.png)

<figcaption>

The effect of poisoning using a subset of 1000 training items.

</figcaption>

</figure>

## Conclusion

The machine learning model is surprisingly robust to wrong data, as it shows the ability to continue learning even when most of the data is wrong. While some bad data is almost unavoidable in a real-world scenario, it's also interesting to observe that worse, but more data can in some situations be preferable to better, but less.

Future work:

- I used a batch size of 256 for training. Would a more standard batch size of 32 worsen training?

- How would poisoning the validation datasets change the plots?

- Would a larger model show different results?


The effect of Dataset Poisoning on Model Accuracy


We often use recurrent neural network architectures for use cases such as streaming, where we make a new prediction each time new information arrives. It's common to give it a long context each time, which repeats work and is therefore computationally wasteful. In this blog post, I will illustrate multiple ways to perform inference step by step using recurrent architectures by remembering the states and therefore removing the need to unnecessarily compute the same steps over and over. My post [Creating a custom training loop in tensorflow](/posts/2020-04-26-creating-a-custom-training-loop-in-tensorflow/) gives a quick start on the [Keras functional API](https://keras.io/guides/functional_api/) which I use for the main code snippets.

## Context

Recurrent Neural Networks (RNNs) are a popular type of neural network which specialises in time series data. Just like our brains are able to interpret a series of images as a movie or a bunch of words as a sentence, so a recurrent neural network is able to remember the past inputs, which affect its output for future inputs. It does this by maintaining a _hidden state_ which is fed forward to the next iteration in a recurrent fashion. The unit that takes the previous state and input as inputs and returns the context is called a _cell._

To improve this basic principle, architectures such as the LSTM (Long Short-Term Memory) improve on the vanilla RNN cell by including a separate _cell state_ and special gates that "remember" and "forget" to and from the cell state. This greatly improves the ability of these networks to remember long sequences. This post will focus on the LSTM cell due to the extra complexity from the second state but can be easily adapted to the other main recurrent architecture used, the GRU cell.

<figure>

![RNN Architectures](/images/rnn_architecture.png)

<figcaption>

Basic architectures of RNN, GRU and LSTM cells, by Saurabh Rathor on [medium](https://medium.com/@saurabh.rathor092/simple-rnn-vs-gru-vs-lstm-difference-lies-in-more-flexible-control-5f33e07b1e57)

</figcaption>

</figure>

## The wasteful approach

The wasteful approach provides the entire context for each time step. To predict timestep _i_, the model receives inputs _\[0..i\]_ as context. Since the hidden state after time step _k, 0 ≤ k < i_ is deterministically the same, repeating the computation is unnecessary.

First, we need to define imports, fix the seed, set the parameters for the models and generate a random input that we will reuse with all the models:

```python
# Imports
import tensorflow as tf
import tensorflow.keras.layers as L
import matplotlib.pyplot as plt
from time import perf_counter

# Fix the seed so it always generates the same X and weights
tf.random.set_seed(0)

# Define parameters
batch_size = 10
seq_length = 200
features = 4
lstm_1_units = 100
lstm_2_units = 200

# Generate a random input for all models
X = tf.random.uniform((batch_size, seq_length, features))
```

The "classic" way. A standard model, as expected.

```python
input1 = L.Input((None, 4))

lstm1_1 = L.LSTM(lstm_1_units, return_sequences=True)
lstm1_1_o = lstm1_1(input1)

lstm2_1 = L.LSTM(lstm_2_units)
lstm2_1_o = lstm2_1(lstm1_1_o)

output_dense_1 = L.Dense(1, activation="sigmoid")
output1 = output_dense_1(lstm2_1_o)

model1 = tf.keras.Model(inputs=input1, outputs=output1)

start1 = perf_counter()
classic_results = []
for i in range(1, len(X[0])):
    result_batch = model1.predict(X[:, 0:i, :])
    classic_results.append(result_batch[0][0])
stop1 = perf_counter()

plt.plot(classic_results)
plt.title(f"Classic approach took {(stop1 - start1):.03f}s")
plt.show()
```

![Classic approach output](/images/stepbystep_classic.png)

To improve the performance, we often restrict the context by only providing _\[i-B..i\]_. This is significantly better computationally, though it also repeats most operations, and restricts the model from having access to as much of the data as potentially useful.

## The stateful model

Keras offers "stateful=True" for LSTM layers which maintain state automatically.  
This is good if you always have the same sequence order in the batch, e.g.,

- you are streaming live data and you are always appending to the same sequence

- you want to evaluate a long sequence without keeping the entire thing in memory

It needs to know its batch size to be stateful, therefore it is specified for Input

```python
input2 = L.Input((None, 4), batch_size=batch_size)

lstm1_2 = L.LSTM(lstm_1_units, return_sequences=True, stateful=True)
lstm1_2_o = lstm1_2(input2)

lstm2_2 = L.LSTM(lstm_2_units, stateful=True)
lstm2_2_o = lstm2_2(lstm1_2_o)

output_dense_2 = L.Dense(1, activation="sigmoid")
output2 = output_dense_2(lstm2_2_o)

model2 = tf.keras.Model(inputs=input2, outputs=output2)

# Copy the weights from the first model
lstm1_2.set_weights(lstm1_1.get_weights())
lstm2_2.set_weights(lstm2_1.get_weights())
output_dense_2.set_weights(output_dense_1.get_weights())

start2 = perf_counter()

stateful_results = []
for i in range(len(X[0]) - 1):
    # Since it's stateful, we can get each time step one by one
    # entire batch, from i->i+1, entire feature list
    result_batch = model2.predict(X[:, i : i + 1, :])
    stateful_results.append(result_batch[0][0])

stop2 = perf_counter()

plt.plot(stateful_results)
plt.title(f"Stateful model took {(stop2 - start2):.03f}s")
plt.show()
```

![Step by step - stateful approach output](/images/stepbystep_stateful.png)

## Step by step inference using the functional API

Maybe items in the batch change, or are not always present  
Manually managing the state is useful as it allows some items in the batch to continue processing  
The stateful approach only allows `reset_states()`, but that clears all of them

using the functional model we can accept multiple inputs which we pass as c,h

This model takes as inputs the c,h for lstm1, lstm2  
returns new c,h for lstm1, lstm2  
An input for each c,h  
And returns each c,h

Management of them can then be done in the prediction loop as required or saved for later.

```python
input3 = L.Input((None, features))
lstm1_3_c_input = L.Input((lstm_1_units,))
lstm1_3_h_input = L.Input((lstm_1_units,))
lstm2_3_c_input = L.Input((lstm_2_units,))
lstm2_3_h_input = L.Input((lstm_2_units,))

lstm1_3 = L.LSTM(lstm_1_units, return_sequences=True, return_state=True)
lstm1_3_o, lstm1_3_h, lstm1_3_c = lstm1_3(
    input3, initial_state=[lstm1_3_h_input, lstm1_3_c_input]
)
lstm2_3 = L.LSTM(lstm_2_units, return_state=True)
lstm2_3_o, lstm2_3_h, lstm2_3_c = lstm2_3(
    lstm1_3_o, initial_state=[lstm2_3_h_input, lstm2_3_c_input]
)

output_dense_3 = L.Dense(1, activation="sigmoid")
output3 = output_dense_3(lstm2_3_o)

model4 = tf.keras.Model(
    inputs={
        "input": input3,
        "lstm1_h": lstm1_3_h_input,
        "lstm1_c": lstm1_3_c_input,
        "lstm2_h": lstm2_3_h_input,
        "lstm2_c": lstm2_3_c_input,
    },
    outputs={
        "output": output3,
        "lstm1_h": lstm1_3_h,
        "lstm1_c": lstm1_3_c,
        "lstm2_h": lstm2_3_h,
        "lstm2_c": lstm2_3_c,
    },
)

lstm1_3.set_weights(lstm1_1.get_weights())
lstm2_3.set_weights(lstm2_1.get_weights())
output_dense_3.set_weights(output_dense_1.get_weights())

# Initialise cell hidden states, default zeros
lstm1_3_h_val = tf.zeros((batch_size, lstm_1_units))
lstm1_3_c_val = tf.zeros((batch_size, lstm_1_units))
lstm2_3_h_val = tf.zeros((batch_size, lstm_2_units))
lstm2_3_c_val = tf.zeros((batch_size, lstm_2_units))

start3 = perf_counter()
manual_func_results = []
for i in range(len(X[0]) - 1):
    ts_res = model4.predict(
        {
            "input": X[:, i : i + 1, :],
            "lstm1_h": lstm1_3_h_val,
            "lstm1_c": lstm1_3_c_val,
            "lstm2_h": lstm2_3_h_val,
            "lstm2_c": lstm2_3_c_val,
        }
    )
    manual_func_results.append(ts_res["output"][0][0])
    lstm1_3_h_val = ts_res["lstm1_h"]
    lstm1_3_c_val = ts_res["lstm1_c"]
    lstm2_3_h_val = ts_res["lstm2_h"]
    lstm2_3_c_val = ts_res["lstm2_c"]
stop3 = perf_counter()

plt.plot(manual_func_results)
plt.title(f"Manual functional model took {(stop3 - start3):.03f}s")
plt.show()
```

![Step by step - functional approach output](/images/stepbystep_functional.png)

## Step by step inference using the subclassed model

```python
class SubclassedModel(tf.keras.Model):
    def __init__(self):
        super().__init__()
        self.lstm1 = L.LSTM(lstm_1_units, return_sequences=True, return_state=True)
        self.lstm2 = L.LSTM(lstm_2_units, return_state=True)
        self.output_dense = L.Dense(1, activation="sigmoid")

    def call(self, inputs):
        lstm1_o, lstm1_h, lstm1_c = self.lstm1(
            inputs["input"], initial_state=[inputs["lstm1_h"], inputs["lstm1_c"]]
        )
        lstm2_o, lstm2_h, lstm2_c = self.lstm2(
            lstm1_o, initial_state=[inputs["lstm2_h"], inputs["lstm2_c"]]
        )
        output_dense = self.output_dense(lstm2_o)
        return {
            "output": output_dense,
            "lstm1_h": lstm1_h,
            "lstm1_c": lstm1_c,
            "lstm2_h": lstm2_h,
            "lstm2_c": lstm2_c,
        }

model4 = SubclassedModel()
model4.build(
    input_shape={
        "input": (batch_size, None, features),
        "lstm1_h": (batch_size, lstm_1_units),
        "lstm1_c": (batch_size, lstm_1_units),
        "lstm2_h": (batch_size, lstm_2_units),
        "lstm2_c": (batch_size, lstm_2_units),
    }
)

model4.lstm1.set_weights(lstm1_1.get_weights())
model4.lstm2.set_weights(lstm2_1.get_weights())
model4.output_dense.set_weights(output_dense_1.get_weights())

lstm1_4_h_val = tf.zeros((batch_size, lstm_1_units))
lstm1_4_c_val = tf.zeros((batch_size, lstm_1_units))
lstm2_4_h_val = tf.zeros((batch_size, lstm_2_units))
lstm2_4_c_val = tf.zeros((batch_size, lstm_2_units))

state = {
    "lstm1_h": lstm1_4_h_val,
    "lstm1_c": lstm1_4_c_val,
    "lstm2_h": lstm2_4_h_val,
    "lstm2_c": lstm2_4_c_val,
}

start4 = perf_counter()
manual_subcl_results = []
for i in range(len(X[0]) - 1):
    state["input"] = X[:, i : i + 1, :]
    state = model4.predict(state)
    manual_subcl_results.append(state["output"][0][0])
stop4 = perf_counter()

plt.plot(manual_subcl_results)
plt.title(f"Subclassed model took {(stop4 - start4):.03f}")
plt.show()
```

![Step by step - subclassed approach output](/images/stepbystep_subclassed.png)

## Ensuring all outputs were identical

```python
assert tf.reduce_all(classic_results == stateful_results)
assert tf.reduce_all(stateful_results == manual_func_results)
assert tf.reduce_all(manual_func_results == manual_subcl_results)
```

## Conclusion

Recurrent architectures are known for being slow, and it's important to notice the speedup obtained even with such a small model. Knowing how to manually manage recurrent hidden states improves the performance of machine learning models used for streaming or deployed in stateless containers


Step by step recurrent neural network inference with Keras


![make beat saber maps using machine learning](/images/beat-saber-screen.jpg)

I love playing Beat Saber, a VR rhythm game in which you slice blocks in time with music. Turning a song into a Beat Saber map requires serious time, dedication and talent. I'll attempt to bypass the requirements by trying to use artificial intelligence to make the Beat Saber maps.

In order to create such a model, I first need to understand the format of beat maps so I can read them for data and also to ultimately create my own. I then need to analyse them so I can gain a better understanding of the feature space and obtain some intuition about what sort of model I can use. Then, I'll need to craft the model, train it, figure out how to evaluate the results, and iterate until something usable hopefully comes out. Finally, I'd like to make the model available for other people to use.

This first post will begin the exploration part of the project by having a look at the data.

The [cookiecutter data science project template](https://drivendata.github.io/cookiecutter-data-science/) is a great starting point for [the project repository](https://github.com/felixgravila/bsabermapmaker). I performed the analysis presented in this post using a Jupyter notebook which can be found [here](https://github.com/felixgravila/bsabermapmaker/blob/master/notebooks/01-investigate-combinations.ipynb).

##### What does the data look like

While having never created a map, playing the game gave me an intuition for what to expect. The first step was to investigate an existing song. Looking into a song directory we see the following structure:

```raw
.
├── Expert.dat
├── Hard.dat
├── Normal.dat
├── Toto\ -\ Africa.egg
├── cover.jpg
└── info.dat
```

The folder contains a jpg cover photo, an egg audio file which contains the music track, and multiple dat files. The info file represents the meta-information about the song in JSON format: the name and artist, but also direct links to the cover image, audio file, and each difficulty map, also stored as JSON dat files.

The maps are also JSON files. Notably, they contain lists such as "\_events", "\_obstacles", and "\_notes". I did not know what events are, so I simply removed them from the list and played the song. A dark scene with no lights confirmed that "\_events" handles the lighting effects. For simplicity, I'll study the notes, since they represent the core of the gameplay. Each note is composed of the following parameters:

```json
{
      "_time": 28.5625,
      "_lineIndex": 2,
      "_lineLayer": 0,
      "_type": 1,
      "_cutDirection": 0
},
```

I used [BeatMapper](https://beatmapper.app/) to create a pattern of notes to help me understand what each parameter means.

```raw
"_time": time (seconds) of the note from the beginning of the song
"_lineIndex": the position on the x-axis of the note, from 0-3
"_lineLayer":  the position on the y-axis of the note, from 0-2
"_type": the hand of the note, 0 = left, 1 = right
"_cutDirection": the direction in which to cut the note. 0-7 are the 7 directions and 8 is the dot-block
```

##### Song analysis

To get some statistics on the data, I used the Jupyter notebook linked in the introduction. Since the notebook is commented, I'll only quickly present my findings. For raw data, I linked my existing library of 500 songs.

The game is designed to handle any lineIndex and lineLayer value. Even so, most Expert+ songs use the base 3x4 grid. I use this to justify simplifying the dataset and therefore the feature space.

<figure>

![make beat saber maps using machine learning](/images/bsheatmap.png)

<figcaption>

Heat map of note positions

</figcaption>

</figure>

Additionally, I used bit-shifting to create a unique 16-bit identifier for each possible simultaneous note pattern. I limited each pattern to at most one block per hand.

![](/images/bsconfigoccurencegraph.png)

This analysis shows that an output of 16 neurons can represent a time step worth of note data. In addition, while the number of theoretically possible combinations is astronomical in size, almost 99% of all occurring configurations are represented by the top 1000 configurations.

##### Conclusion

Investigating data plays an essential role in building a machine learning model. While not a complete analysis yet, I hope it can already be a source of inspiration and illustrate the importance of getting a good idea about the data you're working with. Thank you for reading the first post about my project to use machine learning to make beat saber maps.


make beat saber maps using machine learning (1)


I love flying around in flight simulators. As fun as it is to fly around and try to figure out procedures, it's rarely a _game_ as such. There are no missions, no goals, no penalty for mistakes. [FSEconomy](https://www.fseconomy.net/) is a plugin for flight simulators which adds a layer of gamification by enabling users, among others, to rent and buy aircraft and fly existing missions in order to earn more in-game currency. Users can own small planes and fly them anywhere, but large aircraft are tied to specific missions. Since there is no easy way to find these missions, I decided to spend a bit of time developing a plotting tool for FSEconomy, which displays the missions on a map.

Initially just a simple map with a field for the access key and a drop-down for the plane, I quickly realised I need to be as transparent as possible since people are entrusting their private API keys. I, therefore, made the project open source and added a pop-up modal to explain why it needs your key. I also use a proxy due to CORS which I disclose there.

After launching the site and posting on the FSEconomy Discord, I received good feedback and suggestions for improvement. I quickly implemented the ability to go to the FSEconomy airport page after clicking on the airport, and optional cookies which remember your key and selections for next time.

I deliberately decided to not use any web frameworks for development. Like many others, I'm sick of bloated websites when it's just not necessary. In order not to add to the pool, I kept it simple and only used Bootstrap, JQuery and Leaflet. When I started implementing the functionality to display all missions in a sortable table, I almost regretted my decision. I proceeded to put hours of work into reorganising my code, HTML, and making sure the CSS animations don't break. Sorting is performed by erasing the table and re-adding the data.

##### Screenshots

![Plotting Tool For FSEconomy](/images/fseplot_map.png)

![Plotting Tool For FSEconomy](/images/fseplot_table.png)

You can find the code for the plotting tool for FSEconomy on my [GitHub](https://github.com/felixgravila/fseconomy-plot-missions).


Plotting tool for FSEconomy


In the past, I've covered [how to implement k-means](/posts/2020-05-03-implement-k-means-in-tensorflow/) and [how to cluster MNIST directly](/posts/2020-05-10-clustering-mnist-naive/). Then I showed [a bad way to reduce dimensionality for clustering](/posts/2020-05-17-how-not-to-reduce-dimensionality-for-clustering/). Finally, we [used an autoencoder on MNIST](/posts/2020-05-24-reduce-mnist-dimensionality-using-autoencoders/). In this post, we'll join everything together by performing clustering on the encoding and see how it performs.

<figure>

![Clustering On The Encoding](/images/autoenc-rot-1.gif)

<figcaption>

Encoding and centroids

</figcaption>

</figure>

In the previous post, we've used an autoencoder to encode the MNIST dataset. By constraining all 784 dimensions of the images into the 3 of the bottleneck, we hoped similar digits would be clustered together. We confirmed this since we could see a clear clustering when colour coding by digit.

![Autoencoding output](/images/autoencode_output.png)

It follows that a great idea would be to cluster this encoding as we hope the important features were already extracted. Remember that the goal of this exercise is unsupervised learning. The neural network is able to extract features combining multiple pixels, therefore increasing its power.

##### Autoencoder again

The first part of the code forms the same autoencoder. First some definitions:

```python
import tensorflow as tf
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.patches as mpatches
import numpy as np

# colour list for plotting
colours = ['#00FA9A','#FFFF00','#2F4F4F','#8B0000','#FF4500','#2E8B57','#6A5ACD','#FF00FF','#A9A9A9','#0000FF']

# number of features for the encoded number
encoded_dim = 3

# load MNIST data
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

# scale between 0 and 1
X = tf.constant(x_train/255.0)

# reshape since we're feeding it into a dense layer
X = tf.reshape(X, (-1, 28*28))
```

We redefine the model the same way. It can be trained by compiling and fitting like previous or just load the previous model from the weights file. The model can be loaded completely from the h5 file without having to redefine it. Since we split the autoencoder, we redefine it so we have access to the layers using which we build our encoder and decoder.

```python
# Define the model again so we can load the weights from last time and split into encoder/decoder

# First the encoder
enc_input = tf.keras.layers.Input(shape=(28*28,), name="enc_input")
enc_inner = tf.keras.layers.Dense(28*28, activation="sigmoid", name="enc_dense_1")(enc_input)
enc_inner = tf.keras.layers.Dense(64, activation="sigmoid", name="enc_dense_2")(enc_inner)
enc_inner = tf.keras.layers.Dense(32, activation="sigmoid", name="enc_dense_3")(enc_inner)
enc_output = tf.keras.layers.Dense(encoded_dim, activation="sigmoid", name="enc_output")(enc_inner)

# Then the decoder
dec_inner = tf.keras.layers.Dense(32, activation="sigmoid", name="dec_dense_1")(enc_output)
dec_inner = tf.keras.layers.Dense(64, activation="sigmoid", name="dec_dense_2")(dec_inner)
dec_output = tf.keras.layers.Dense(28*28, activation="sigmoid", name="dec_output")(dec_inner)

# Define the whole autoencoder from enc_input to dec_output
autoencoder = tf.keras.Model(inputs=enc_input, outputs=dec_output, name="autoencoder")

# The encoder stops at enc_output
encoder = tf.keras.backend.function(enc_input, enc_output)
# decoder from enc_output to dec_output
decoder = tf.keras.backend.function(enc_output, dec_output)

autoencoder.load_weights(f"autoenc_{encoded_dim}.h5")
```

To have saved the model in the last post, simply add the following after training it:

```python
autoencoder.save(f"autoenc_{encoded_dim}.h5")
```

##### K-means

We can use the K-means code from the clustering MNIST post.

```python
# define number of clusters
clusters_n = 10

pred = encoder(X)

centroids = tf.slice(tf.compat.v1.random_shuffle(pred), [0, 0], [clusters_n, -1])
points_expanded = tf.expand_dims(pred, 0)

@tf.function
def update_centroids(points_expanded, centroids):
    centroids_expanded = tf.expand_dims(centroids, 1)
    distances = tf.subtract(centroids_expanded, points_expanded)
    distances = tf.square(distances)
    distances = tf.reduce_sum(distances, 2)
    assignments = tf.argmin(distances, 0)

    means = []
    for c in range(clusters_n):
        eq_eq = tf.equal(assignments, c)
        where_eq = tf.where(eq_eq)
        ruc = tf.reshape(where_eq, [1,-1])
        ruc = tf.gather(pred, ruc)

        ruc = tf.reduce_mean(ruc, axis=[1])
        means.append(ruc)
        new_centroids = tf.concat(means, 0)

    return new_centroids, assignments

old_centroids = centroids
while True:
    centroids, assignments = update_centroids(points_expanded, centroids)
    if tf.reduce_all(centroids == old_centroids):
        break
    old_centroids = centroids

# classify points using trained centroids
# same code as for update_centroids, but only returns the argmin
def get_assignments(centroids, y_pred):
    points_expanded = tf.expand_dims(y_pred, 0)
    centroids_expanded = tf.expand_dims(centroids, 1)
    distances = tf.subtract(centroids_expanded, points_expanded)
    distances = tf.square(distances)
    distances = tf.reduce_sum(distances, 2)
    assignments = tf.argmin(distances, 0)
    return assignments
```

We can then plot the representation with the centroids.

```python
# Plot the encoded representations and the centroids
if encoded_dim == 3:
    num_to_plot = 10000
    pred = encoder(X[:num_to_plot])
    fig = plt.figure(figsize=(10,10))
    ax = fig.add_subplot(111, projection='3d')

    ax.scatter(pred[:,0], pred[:,1], pred[:,2], zorder=1, color=[colours[y] for y in y_train[:num_to_plot]])
    ax.plot(centroids[:, 0], centroids[:, 1], centroids[:, 2], "kX", markersize=20, zorder=1000)
    mpc = []
    for i in range(10):
        mpatch = mpatches.Patch(color=colours[i], label=i)
        mpc.append(mpatch)
    plt.legend(handles=mpc)
```

![Clustering On The Encoding](/images/autoenc.png)

Since we have both an encoder and decoder, and the centroids are in the feature space of the encoding, we can use the decoder to see what the centroids look like.

```python
# Get reconstructed numbers for centroids
reconst = decoder(centroids)

fig = plt.figure(figsize=(10, 10))
for i, c in enumerate(reconst):
    ax = fig.add_subplot(3, 4, i+1)
    plt.imshow(tf.reshape(c, (28, 28)), cmap="gray_r")
```

![Clustering On The Encoding](/images/centroids_decoded.png)

Finally, let's have a look at the purity

```python
def calc_purity(labels, assignments):
    d = np.zeros((clusters_n, clusters_n), dtype="int32")
    for l, a in zip(labels, assignments):
        d[a][l] += 1

    purity_per_class = d.max(1)/d.sum(1)
    # some are NaN
    purity_per_class = purity_per_class[~np.isnan(purity_per_class)]
    return np.mean(purity_per_class)

# Perform prediction on the dataset to get the intermediate representation
predict_batch_size = 10000
predict_count = len(X)

m = []
for i in range(0, predict_count, predict_batch_size):
    m.append(encoder(X[i:i+predict_batch_size]))
res = tf.concat(m, 0)

assignments = get_assignments(centroids, res)

calc_purity(y_train[:predict_count], assignments)
```

```
0.6988619101877526
```

Already better than the 0.62 obtained by directly clustering on the pixels.

##### Conclusion

I hope this post illustrated the power of clustering on the encoding by using an autoencoder. Beside being able to use the power of a neural network to extract complex features, the decoder enables us to directly reconstruct the centroids, giving us a better intuition of what they are doing, which can increase trust in the model.


Clustering on encoding


Dimensionality reduction is useful in compressing the representation of your data. In my last post, [how not to reduce dimensionality for clustering](/posts/2020-05-17-how-not-to-reduce-dimensionality-for-clustering/), I show a neural network that reduces the 784 dimensions of the MNIST dataset into 3, which are constrained by the k-means algorithm to capture the essence of the data. This week we'll be looking to reduce the dimensionality of MNIST using autoencoders since it is a more common approach.

Autoencoders are a type of neural network which constrain the output to be the same as the input. The information passes through a bottleneck of low dimensionality. This has the effect of letting the network learn how to embed all the relevant information into a smaller representation.

![reduce dimensionality autoencoders](/images/autoenc-anim.gif)

##### The model

Let's go through the code in order to see how it's done. As usual, we start with the imports and some definitions. Check out my shapes posts if you're confused about reshaping.

```python
import tensorflow as tf
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.patches as mpatches

# colour list for plotting
colours = ['#00FA9A','#FFFF00','#2F4F4F','#8B0000','#FF4500','#2E8B57','#6A5ACD','#FF00FF','#A9A9A9','#0000FF']

# number of features for the encoded number
encoded_dim = 3

# load MNIST data
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

# scale between 0 and 1
X = tf.constant(x_train/255.0)

# reshape since we're feeding it into a dense layer
X = tf.reshape(X, (-1, 28*28))
```

The autoencoder consists of two parts: the encoder and the decoder. We use the decoder to handle dimensionality reduction and the decoder to recreate the original data. We'll build a functional model in order to be able to split the model into pieces.

```python
# First the encoder
enc_input = tf.keras.layers.Input(shape=(28*28,))
enc_inner = tf.keras.layers.Dense(28*28, activation="sigmoid")(enc_input)
enc_inner = tf.keras.layers.Dense(64, activation="sigmoid")(enc_inner)
enc_inner = tf.keras.layers.Dense(32, activation="sigmoid")(enc_inner)
enc_output = tf.keras.layers.Dense(encoded_dim, activation="sigmoid")(enc_inner)

# Then the decoder
dec_inner = tf.keras.layers.Dense(32, activation="sigmoid")(enc_output)
dec_inner = tf.keras.layers.Dense(64, activation="sigmoid")(dec_inner)
dec_output = tf.keras.layers.Dense(28*28, activation="sigmoid")(dec_inner)

# Define the whole autoencoder from enc_input to dec_output
autoencoder = tf.keras.Model(inputs=enc_input, outputs=dec_output)

# The encoder stops at enc_output
encoder = tf.keras.backend.function(enc_input, enc_output)

# compile the model so we penalise based on mean squared error between pixels
autoencoder.compile(optimizer="adam", loss="mse")
```

I like using EarlyStopping so I don't waste time training. It works by stopping training if the model accuracy does not improve over more than 10 epochs. We let it fit for a maximum of 500 epochs, but in practice, it stops before 100. Notice how we fit using X both as input and labels.

```python
es = tf.keras.callbacks.EarlyStopping(monitor='loss', patience=10, min_delta=0.0005)
autoencoder.fit(X, X, epochs=500, callbacks=[es])
```

##### How Does it perform?

We can plot both the original and the autoencoded images to see how well the model performs. The higher the encoded dimension, the more accurate the result. Let's see how encoding into 3 dimensions compares to 10 dimensions.

```python
# Plot autoencoder outputs compared to inputs
def plot_examples(autoencoder, X):
    fig = plt.figure(figsize=(5,2*len(X)))
    output = autoencoder(X)
    X = tf.reshape(X, (-1, 28, 28))
    output = tf.reshape(output, (-1, 28, 28))

    for j,(i,o) in enumerate(zip(X, output)):
        ax = fig.add_subplot(len(X), 2, 2*j+1)
        ax.imshow(i)
        ax = fig.add_subplot(len(X), 2, 2*j+2)
        ax.imshow(o)

plot_examples(autoencoder, X[:9])
```

<figure>

![reduce dimensionality autoencoders](/images/autoenc-mnist.png)

<figcaption>

Left: 3 dimensions, Right: 10 dimensions

</figcaption>

</figure>

Using 10 dimensions yields more detailed outputs, but the 3 dimension model isn't bad at all. However, The bonus to using 3 dimensions is being able to easily plot the intermediate representation as a 3D plot.

<figure>

![reduce dimensionality autoencoders](/images/autoencode_output.png)

<figcaption>

Intermediate representation of the MNIST dataset in 3 dimensions

</figcaption>

</figure>

It is impressive seeing how dots representing the same digit continue to be close together even in 3 dimensions. I imagine there will be little surprise about the theme of my next post.

Thanks for reading! I hope this post was interesting since autoencoders are a cool type of neural network. As an example, we can reduce dimensionality using autoencoders to compress data or to extract important features.


Reduce MNIST dimensionality using autoencoders


In my previous post, I followed up on [my k-means tutorial](/posts/2020-05-03-implement-k-means-in-tensorflow/) by [applying it to cluster MNIST](/posts/2020-05-10-clustering-mnist-naive/). We're used to reading digits, so the centroids made perfect sense to us since plotting them just looked like a smudged number. The dataset even clustered pretty well since all digits are centred and look similar enough that decisions can directly be made on individual pixels. However, if we imagine those 784 pixels to be sensor readings, it would likely be really hard to make sense of them. Additionally, many situations would benefit from some nonlinear transformations to better capture the relationships between features. What we want, therefore, is to reduce the original dimensionality, so clustering becomes more manageable.

In this post, we'll try to pass the data through a neural network which outputs in three dimensions. We then use K-means directly as a loss function, trying to get it to cluster points close to their assigned centroids and far from the others. This method is supposed to fail. However, I hope to use this as an entry point to some more interesting methods. Essentially, I also just thought I'd show it off since I haven't seen it presented before.

<figure>

![reduce dimensionality for clustering](/images/knn-dim-red-anim.gif)

<figcaption>

condensed representations of the digits moving in the feature space while training

</figcaption>

</figure>

#### Build the model

We do the preliminaries and create our model. I made a model with two CNN layers and MaxPooling into a 3 unit dense output. In order to plot it in 3 dimensions, I chose 3. However, I tried it with 10 as well and it didn't perform any better.

```python
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import random
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.patches as mpatches
import math

colours = ['#00FA9A','#FFFF00','#2F4F4F','#8B0000','#FF4500','#2E8B57','#6A5ACD','#FF00FF','#A9A9A9','#0000FF']

# we want to split into 10 clusters, one for each digit
clusters_n = 10

# load MNIST data
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

# scale between 0 and 1
X = tf.constant(x_train/255.0)

class MyModel(tf.keras.Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.cnn_1 = tf.keras.layers.Conv2D(filters=32, kernel_size=(3,3), activation="relu")
        self.mp_1 = tf.keras.layers.MaxPool2D(pool_size=2)
        self.cnn_2 = tf.keras.layers.Conv2D(filters=16, kernel_size=(5,5), activation="relu")
        self.mp_2 = tf.keras.layers.MaxPool2D(pool_size=2)
        self.flatten = tf.keras.layers.Flatten()
        self.dense_1 = tf.keras.layers.Dense(12, activation="relu")
        self.dense_2 = tf.keras.layers.Dense(3)

    def call(self, x):
        inner = self.cnn_1(x)
        inner = self.mp_1(inner)
        inner = self.cnn_2(inner)
        inner = self.mp_2(inner)
        inner = self.flatten(inner)
        inner = self.dense_1(inner)
        inner = self.dense_2(inner)
        return inner
```

We need to pass the centroids as parameters both since we want to keep updating the same ones, and since GradientTape doesn't like random initialisations anyways. The new function is half of the code for centroid updating, which we'll use just to perform prediction once we have the model outputs and the trained centroids.

```python
@tf.function
def update_centroids(orig_points, points_expanded, centroids):
    centroids_expanded = tf.expand_dims(centroids, 1)
    distances = tf.subtract(centroids_expanded, points_expanded)
    distances = tf.square(distances)
    distances = tf.reduce_sum(distances, 2)
    assignments = tf.argmin(distances, 0)

    means = []
    for c in range(clusters_n):
        eq_eq = tf.equal(assignments, c)
        where_eq = tf.where(eq_eq)
        ruc = tf.reshape(where_eq, [1,-1])
        ruc = tf.gather(orig_points, ruc)

        ruc = tf.reduce_mean(ruc, axis=[1])
        means.append(ruc)
        new_centroids = tf.concat(means, 0)

    return new_centroids, assignments, distances

def do_kmeans(centroids, y_pred):
    points_expanded = tf.expand_dims(y_pred, 0)

    old_centroids = centroids
    i = 0
    while True and i < 50:
        i+=1
        centroids, assignments, distances = update_centroids(y_pred, points_expanded, centroids)
        if tf.reduce_all(tf.equal(old_centroids, centroids)):
            break
        old_centroids = centroids

    return centroids, assignments, distances

# classify points using trained centroids
# same code as for update_centroids, but only returns the argmin
def get_assignments(centroids, y_pred):
    points_expanded = tf.expand_dims(y_pred, 0)
    centroids_expanded = tf.expand_dims(centroids, 1)
    distances = tf.subtract(centroids_expanded, points_expanded)
    distances = tf.square(distances)
    distances = tf.reduce_sum(distances, 2)
    assignments = tf.argmin(distances, 0)
    return assignments
```

#### Training

The training loop is fairly standard. I batch the data so it fits in the GPU. I'm counting on the fact that 5000 digits are varied enough to not be detrimental to training, but I might be wrong.

```python
EPOCHS = 1000
BATCH_SIZE = 5000
LR = 0.0001

# patience is the number of epochs until it stops training if no loss improvements
patience = 10
waited = 0

# initialise our model
model = MyModel()

# use adam as optimiser
adam = tf.keras.optimizers.Adam(learning_rate=LR)

# initialise a large initial loss on which to improve
prev_best_loss = 1e10

# variable to save prev best weights in
# we want to load it back after patience runs out
prev_weights = None

# define centroids which we'll keep updating
g_centroids = None

for e in range(EPOCHS):
    # shuffle data each time
    shuffled_data = tf.random.shuffle(X)
    # we'll batch it so it fits in GPU memory
    batched_data = tf.reshape(shuffled_data, (-1, BATCH_SIZE, 28, 28, 1))
    print(f"Epoch {e+1}", end="")

    # variable to keep track of total epoch loss
    tot_epoch_loss = 0
    for idx, batch in enumerate(batched_data):
        with tf.GradientTape() as g:
            # predict
            output = model(batch)
            # take first clusters_n outputs as initialisation for centroids
            if g_centroids is None:
                g_centroids = output[:clusters_n]

            # now we do k-means on the output of the model
            g_centroids, assignments, distances = do_kmeans(g_centroids, output)

            # compute the sum of minimum distances to a centroid
            # in other words, the sum of distances between all points and their assigned centroid
            # we want to minimise this to get them as tight as possible
            dis_to_c = tf.reduce_sum(tf.reduce_min(distances, 0))

            # we then want to also maximise the distance from all points to all other centroids
            # otherwise the model would clump everything together
            # we can get this by summing all distances and then subtracting the smallest ones
            dis_to_all = tf.reduce_sum(distances)
            dis_to_others = dis_to_all - dis_to_c

            loss = dis_to_c/float(BATCH_SIZE) + 1/(dis_to_others/float(BATCH_SIZE))

        # I had the loss go to NaN before
        # pretty sure I fixed it but I'm not risking it
        if math.isnan(loss):
            print("N", end="")
            continue

        tot_epoch_loss += loss
        gradients = g.gradient(loss, model.trainable_variables)

        print(f".", end="")
        adam.apply_gradients(zip(gradients, model.variables))

    print(f"Epoch loss {tot_epoch_loss:.2f} ", end="")

    # if best loss save the weights and reset patience
    if tot_epoch_loss < prev_best_loss:
        prev_weights = model.get_weights()
        prev_best_loss = tot_epoch_loss
        waited = 0
    else:
        waited += 1

    print(f"Patience {waited}/{patience}")

    # if no more patience load best weights and quit
    if waited >= patience:
        model.set_weights(prev_weights)
        break
```

Training takes a while, depending on your hardware.

#### Results

We can make a function that plots the numbers in the new feature space. We predict a number of centroids, scatter them with the corresponding colour, then plot the centroids.

```python
def plot_dataset(X, y, model, centroids):
    # Perform prediction on the dataset to get the intermediate representation
    predict_batch_size = 10000
    predict_count = 10000

    m = []
    for i in range(0, predict_count, predict_batch_size):
        m.append(model(tf.reshape(X[i:i+predict_batch_size], (-1, 28, 28, 1))))
    res = tf.concat(m, 0)

    # scatter the points in the embedded feature space and the centroids
    fig = plt.figure(figsize=(20,20))
    ax = fig.add_subplot(111, projection='3d')

    ax.scatter(res[:,0], res[:,1], res[:,2], zorder=1, color=[colours[y] for y in y_train[:predict_count]])
    ax.plot(centroids[:, 0], centroids[:, 1], centroids[:, 2], "kX", markersize=20, zorder=1000)

    # The following just handles printing the colours in the legend
    mpc = []
    for i in range(10):
        mpatch = mpatches.Patch(color=colours[i], label=i)
        mpc.append(mpatch)
    plt.legend(handles=mpc)

plot_dataset(X, y_train, model, g_centroids)
```

![reduce dimensionality for clustering](/images/not-mnist-clustering.png)

It seems it did something and managed to somehow cluster a few of the digits together. Finally, let's calculate purity.

```python
def calc_purity(labels, assignments):
    d = np.zeros((clusters_n, clusters_n), dtype="int32")
    for l, a in zip(labels, assignments):
        d[a][l] += 1

    purity_per_class = d.max(1)/d.sum(1)
    # some are NaN
    purity_per_class = purity_per_class[~np.isnan(purity_per_class)]
    return np.mean(purity_per_class)

assignments = get_assignments(g_centroids, res)
calc_purity(y_train, assignments)
```

```raw
0.3153489094910568
```

#### Conclusion

It's much worse than last time. Oh well. I imagine it's part of the reason why nobody uses this method.

In any case, I hope this at least showed an uncommon way to reduce dimensionality for clustering by using k-means directly as a loss function. Stick around, since I hope to be able to illustrate some better ways of doing this in the following few weeks. As usual, please let me know if you find any bugs in my code or suggestions for improvements.


How not to reduce dimensionality for clustering


This is the beginning of a series in which we'll explore ways to perform clustering. In particular, we'll perform clustering on the MNIST dataset, since it's a dataset of high dimensionality which is easy to understand. In my previous post, '[How to implement the K-means algorithm in Tensorflow](/posts/2020-05-03-implement-k-means-in-tensorflow/)', I explain [the code of Sergey Kovalev, Sergei Sintsov, and Alex Khizhniak](https://www.altoros.com/blog/using-k-means-clustering-in-tensorflow/) line by line, and expand it a bit. Understanding K-means will be important for this series, but skipping the implementation is fine.

Clustering is a type of machine learning that finds patterns in data without being told what to do. It's very useful in situations where we want to separate a dataset of which we do not know much. The MNIST dataset is great for illustrating the point: it consists of 70,000 images of 28x28 pixels depicting handwritten digits.

<figure>

![8](/images/mnist8.png)

<figcaption>

number 8 depicted in the MNIST dataset

</figcaption>

</figure>

MNIST is a labeled dataset. For each image, we get a label that tells us what the depicted number is supposed to be. Using this information we could easily create a supervised learning model such as a CNN, and reach very high accuracies. For the purposes of this series let's assume we do not have access to the labels. Instead of 28x28 pixels, let's imagine they are 784 individual sensor readings. Is it possible to get a machine learning model to designate a class to each number without any labels?

K-means does exactly this. The only parameter it requires, K, tells it how many classes to find. It is a simple algorithm with an easy to understand loop. First, it assigns K random centroids (items in the feature space of the original data). Then, until it converges, it assigns the closest centroid to each data point, then moves the centroid to the mean of all assigned data points. K-means always converges, though not necessarily to a global minimum. Make sure to read my other [post](/posts/2020-05-03-implement-k-means-in-tensorflow/) for a rundown of the algorithm, since I'll be skipping over the details in this one.

#### Direct implementation on the feature space

Let's adapt the algorithm to do some clustering on MNIST. As a first example to show its problems, let's directly cluster on the original feature space. For us this means 784 dimensions. The only thing we need to do is flatten the images. If shapes confuse you, I have a few posts on that ([intro](/posts/2020-04-07-shapes-an-introduction/) and [closer look](/posts/2020-04-12-a-closer-look-at-shapes/)).

```python
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import random
​
# we want to split into 10 clusters, one for each digit
clusters_n = 10
​
# load MNIST data
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
​
# flatten the digits into arrays of 784 pixels
flattened = x_train.reshape(-1, 28*28)/255.0
flattened = tf.constant(flattened)
```

We now have data exactly as we needed it in the last post. Let's run the same algorithm.

```
centroids = tf.slice(tf.compat.v1.random_shuffle(flattened), [0, 0], [clusters_n, -1])
points_expanded = tf.expand_dims(flattened, 0)

@tf.function
def update_centroids(points_expanded, centroids):
  centroids_expanded = tf.expand_dims(centroids, 1)
  distances = tf.subtract(centroids_expanded, points_expanded)
  distances = tf.square(distances)
  distances = tf.reduce_sum(distances, 2)
  assignments = tf.argmin(distances, 0)

  means = []
  for c in range(clusters_n):
    eq_eq = tf.equal(assignments, c)
    where_eq = tf.where(eq_eq)
    ruc = tf.reshape(where_eq, [1,-1])
    ruc = tf.gather(flattened, ruc)

    ruc = tf.reduce_mean(ruc, axis=[1])
    means.append(ruc)
    new_centroids = tf.concat(means, 0)

  return new_centroids, assignments

old_centroids = centroids
while True:
  centroids, assignments = update_centroids(points_expanded, centroids)
  if tf.reduce_all(centroids == old_centroids):
    break
  old_centroids = centroids
```

We now have both our centroids and an assignment for each image. We can make a simple function to show us a few example pictures from each class. Since the algorithm had no idea what is what, the classes do not necessarily match the number, unless your random picks were very lucky.

```
def examples_of(number, rows=5, cols=5):
    fig = plt.figure(figsize=(7,7))
    ax = []
    i=0
    while len(ax) < rows*cols:
        if assignments[i] == number:
            ax.append(fig.add_subplot(rows, cols, len(ax)+1))
            plt.imshow(x_train[i], cmap="gray_r")
        i+=1

examples_of(8)
```

![clustering MNIST](/images/minst_2s.png)

I "coincidentally" picked a great example! Completely unsupervised, K-means has done an amazing job in clustering all number 2's together. Highly impressive, since we never even told it what the images contain. Sadly, it doesn't cluster most other numbers this well. Let's see it get confused by the similar curves of 8 and 3:

![clustering MNIST](/images/minst_38.png)

It especially struggles when dealing with the numbers which have a straight line, like 4,7,9. The following shows three classes which it matched to very similar data.

![clustering MNIST](/images/mnist479.png)

#### Insight

Here's something cool. The centroids exist in the same feature space as the input data, so we can print them just the same! Let's see what each class looks like.

```python
reshaped_centroids = tf.reshape(centroids, (clusters_n, 28, 28))
fig = plt.figure(figsize=(15,8))
ax = []

for i in range(clusters_n):
    ax.append(fig.add_subplot(2, 5, len(ax)+1))
    plt.imshow(tf.reshape(reshaped_centroids[i], (28,28)), cmap="gray_r")
```

<figure>

![centroids](/images/centroids.png)

<figcaption>

the centroids

</figcaption>

</figure>

Looking at them it's pretty clear what's happening. While unique shapes such as 0, 1, and 2 were easy to separate, it really blurs the lines for similar ones. It clearly merges 8 and 3 together, and the 9 looks more like a 7 or a 4.

Since it operates on individual pixels, the algorithm has no clever way of grouping shapes and making sense of the data. Just like a simple dense layer it assigns weights to each pixel, but struggles with difficult decision surfaces.

#### Purity

A popular metric to measure clustering performance is purity. It measures how much of a single class is present in each cluster. A purity of 1 means that each cluster contains a single class. Let's implement it for future reference.

```
def calc_purity(labels, assignments):
    # initialise a square matrix of size clusters_n with zeros
    # in it we'll keep track of the true label distribution per cluster
    d = np.zeros((clusters_n, clusters_n), dtype="int32")
    for l, a in zip(labels, assignments):
        # increase the number of label l assigned to cluster a
        d[a][l] += 1

    # divide highest label count per cluster by the total number of points assigned to it
    # the final score is the mean over all clusters
    return np.mean(d.max(1)/d.sum(1))

purity = calc_purity(y_train, assignments)
print(purity)
```

```raw
0.6256766873889321
```

#### conclusion

Hopefully, this post managed to show a simple way of applying the K-means clustering algorithm on the MNIST dataset. If your dataset has a low dimensionality and can be more easily linearly separated, this would be enough for your needs. In the next posts in this series, I hope to investigate methods of dimensionality reduction in order to improve our results.


Clustering MNIST - naïve


The K-means algorithm is a simple clustering algorithm. It is an unsupervised learning algorithm, therefore it automatically finds structure in unordered data. Since TensorFlow does not have a native implementation, it's useful to know the tips and tricks to implement K-means.

I'm hoping to use K-means in an upcoming post, so I want to explain it to anyone who might be unfamiliar. Additionally, make sure to read my posts about shapes [intro](/posts/2020-04-07-shapes-an-introduction/) and [closer look](/posts/2020-04-12-a-closer-look-at-shapes/) if you're unfamiliar with them since this algorithm makes heavy use of understanding the concept.

K-means is a simple algorithm once you understand it. Its idea is to instantiate a centroid (a point in the same feature space as the data) for each expected class. Following this, iteratively move them towards the coordinates that best represent the respective class. Predicting what class a new data point belongs to is as simple as finding the closest centroid in the feature space.

The algorithm begins by initialising k centroids. Conventionally they will be set as k random data points. The iterative algorithm runs as follows:

```raw
initialise k centroids using k random datapoints
while not converged:
    assign each data point to its closest centroid based on distance
    for each centroid:
       compute the mean between all data points assigned to it
       update the centroid so that its coordinates are the previously calculated mean
```

As simple as the algorithm is, it's not trivial to take advantage of the power of tensorflow in order to efficiently compute it. The following is code by Sergey Kovalev, Sergei Sintsov, and Alex Khizhniak from their post, [Implementing k-means Clustering with TensorFlow](https://www.altoros.com/blog/using-k-means-clustering-in-tensorflow/), which I have expanded with step-by-step comments explaining each line. Additionally, I am creating three random uniform distributions to give k-means something more illustrative to predict. Finally, I improve the plotting by allowing both 2D and 3D visualisations, and by drawing a line showing the paths of the centroids.

```python
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import BASE_COLORS
import random

seed = 0
# set the random seed so numpy always generates the same random data
np.random.seed = seed

points_n = 150
clusters_n = 3
dimensions = 3

batch = tf.constant([])

points = []
for c in range(clusters_n):
  tf.random.set_seed(5)
  # get a random uniform distribution, and multiply by 3 so it's bigger on the graph
  dist = tf.multiply(tf.random.uniform((points_n//clusters_n, dimensions), seed=seed), 3)
  # shift it
  rand_shift = np.random.uniform(0, 10, (1, dimensions))
  dist_adj = tf.add(dist, rand_shift)
  points.append(dist_adj)

# concatenate the points list as a tensor
points = tf.concat(points, axis=0)

# plot in 2d if 2d, else the first 3 dimensions in 3d
fig = plt.figure()
if dimensions == 2:
  ax = fig.add_subplot(111)
  ax.scatter(points[:,0], points[:,1])
elif dimensions > 2:
  ax = fig.add_subplot(111, projection='3d')
  ax.scatter(points[:,0], points[:,1], points[:,2])

```

![data points before clustering](/images/k_means_data.png)

```python
# initialise centroids as coords of first 3 points
centroids = tf.slice(tf.compat.v1.random_shuffle(points), [0, 0], [clusters_n, -1])

# expand the dim for convenience later in tf.subtract
# points_expanded dims (1, points_n, dimension)
points_expanded = tf.expand_dims(points, 0)

@tf.function
def update_centroids(points_expanded, centroids):
  # expand the second dimension, again, for convenience in tf.subtract
  # centroids_expanded dims (clusters_n, 1, dimension)
  centroids_expanded = tf.expand_dims(centroids, 1)

  # due to broadcasting, tensorflow broadcasts the 1-sized dimensions to fit the other tensor
  # I'll use dimension=2 and centroids=3 in the following for clarity:
  #   points_expanded     (1, points_n, 2)
  #   centroids_expanded  (3, 1,        2)
  # points_expanded is stretched 3 times since its first dimension is 1:
  #   points_expanded     (3, points_n, 2) # 3 of the same
  #   centroids_expanded  (3, 1,        2)
  # centroids is stretched points_n times in the same way
  #   points_expanded     (3, points_n, 2)
  #   centroids_expanded  (3, points_n, 2) # now we have points_n of each centroid
  #
  # exactly what we want! For each centroid, we subtract it from all points
  # we are left with clusters_n tensors, each of shape (points_n, dimensions)
  distances = tf.subtract(centroids_expanded, points_expanded)
  # then we square it
  distances = tf.square(distances)
  # finally, we want the cartesian distance, so we add the two squares together
  # reduce_sum reduces the given dimension (2 is last) by summing
  distances = tf.reduce_sum(distances, 2)

  # argmin returns the index of the minimum number
  # found on the given axis (first)
  # in other words, the closest centroid
  assignments = tf.argmin(distances, 0)

  means = []
  # for each cluster
  for c in range(clusters_n):
    # tensor of same shape as assignments where value=True if
    # assignments value == c
    # if it is closest to the centroid in question
    eq_eq = tf.equal(assignments, c)

    # tensor that only contains the indeces from eq_eq that were True
    # [True, False, False, True] -> [[0],[3]]
    where_eq = tf.where(eq_eq)

    # reshapes (matches_found, 1) to (1,matches_found)
    # It can be rewritten to reshape to (-1,)
    # then make sure to reduce_mean on axis 0
    # the extra dimension is to match the shape of centroids
    ruc = tf.reshape(where_eq, [1,-1])

    # gets the points by the indices previously found
    ruc = tf.gather(points, ruc)

    # gets the averages for each dimension of all points
    ruc = tf.reduce_mean(ruc, axis=[1])
    # we have the values for the new centroid
    # add it to the list
    means.append(ruc)

  # concatenate the centroids in the list to one tensor
  new_centroids = tf.concat(means, 0)

  return new_centroids, assignments

fig = plt.figure()
if dimensions == 2:
  ax = fig.add_subplot(111)
elif dimensions > 2:
  ax = fig.add_subplot(111, projection='3d')

# k-means converges, let's go until then
# save the old centroid
old_centroids = centroids
while True:
  # perform one step
  centroids, assignments = update_centroids(points_expanded, centroids)
  # check if the old centroids are identical to the new ones
  # if yes, we are done
  if tf.reduce_all(centroids == old_centroids):
    break

  # print a line from the old centroid to new so it's pretty
  for o,c,k in zip(old_centroids, centroids, BASE_COLORS):
    if dimensions == 2:
      ax.plot([o[0], c[0]], [o[1], c[1]], c=k)
    elif dimensions > 2:
      ax.plot([o[0], c[0]], [o[1], c[1]], [o[2], c[2]], c=k)

  # save the current centroids for future comparison
  old_centroids = centroids

if dimensions == 2:
  # plot all the dots colored as their final assignments
  ax.scatter(points[:, 0], points[:, 1], c=assignments, s=50, alpha=0.5)
  # plot the final centroids as an X
  ax.plot(centroids[:, 0], centroids[:, 1], 'kx', markersize=15)
elif dimensions > 2:
  ax.scatter(points[:, 0], points[:, 1], points[:, 2], c=assignments, s=50, alpha=0.5)
  ax.plot(centroids[:, 0], centroids[:, 1], centroids[:, 2], 'kx', markersize=15)
plt.show()
```

![training of a k-means algorithm](/images/k_means_trained.png)

#### Conclusion

I hope this post helped you understand the details of how to implement K-means in Tensorflow. I'm planning to put it to good use soon, so make sure to check it out!


How to implement the K-means algorithm in Tensorflow


Sticking to the [sequential model](https://keras.io/getting-started/sequential-model-guide/) limits the freedom you have with regards to creating models in Tensorflow. In this post, I'll start by showing a simple sequential model. I'll then convert it into a [functional model](https://keras.io/getting-started/functional-api-guide/). Finally, I'll use [GradientTape](https://www.tensorflow.org/api_docs/python/tf/GradientTape) to calculate all gradients and control the full train loop, all to improve the possibilities in creating your models.

I'll be using a simple feed-forward network to illustrate the examples. Let's start with the sequential model. To keep the number of weights small, I'll add two layers with one neuron each. Finally, the model will be trained to predict the same number as was input, and I'll generate random data. If you're unfamiliar with shapes, I have a few posts on this topic: [intro](/posts/2020-04-07-shapes-an-introduction/) and [a closer look](/posts/2020-04-12-a-closer-look-at-shapes/).

#### Sequential Model

```python
import tensorflow as tf

# define the sequential model
# it contains two dense layers with one neuron each
model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, input_shape=(1,)),
    tf.keras.layers.Dense(1)
])
# compile the model in order to define the loss function
model.compile(loss="mse")

# define the data as 100000 random numbers
data = tf.random.uniform((100000,1))
# train the model to predict the same number
model.fit(x=data, y=data, epochs=2)

# print the weights
for l in model.weights:
    print(f"{l.name}: {l.numpy()}")
print(model.weights[0].name)
```

```raw
Train on 100000 samples
Epoch 1/2
100000/100000 [==============================] - 2s 17us/sample - loss: 0.0365
Epoch 2/2
100000/100000 [==============================] - 2s 15us/sample - loss: 3.4097e-06
dense/kernel:0: [[-0.6452949]]
dense/bias:0: [0.00095751]
dense_1/kernel:0: [[-1.5518339]]
dense_1/bias:0: [0.00247312]
dense/kernel:0
```

We can see that the input weights for the two neurons are -0.6 and -1.55, respectively, and the biases are 0.0009 and 0.002. As humans, we can immediately tell that we could make the perfect model for the job. We can set both neuron weights to 1 and the biases to 0. Our model did the right thing in keeping biases small. The input weights it picked are also valid, since -0.6452949\*-1.5518339=1.0013, which is close enough to 1.

Let's notice how much Tensorflow manages for us. We haven't set the batch size up, it defaults to 32. The fit function finds the relevant function based on the "mse" string we gave it, obtains a prediction from the input, calculates the loss and backpropagates the error.

#### Functional Model

The functional model gives us more flexibility in accessing the layers. Among other benefits, it allows us to define multiple inputs and outputs. We can try to force both neurons to learn weights close to 1 by constraining the output of the first layer to also be close to the ground truth.

```
import tensorflow as tf

a = tf.keras.layers.Input(shape=(1,))
dense_1 = tf.keras.layers.Dense(1)(a)
dense_2 = tf.keras.layers.Dense(1)(dense_1)

model = tf.keras.Model(inputs=a, outputs=[dense_1, dense_2])

model.compile(loss="mse")

data = tf.random.uniform((100000,1))
model.fit(x=data, y=[data, data], epochs=2)

for l in model.weights:
    print(f"{l.name}: {l.numpy()}")
print(model.weights[0].name)
```

```raw
Train on 100000 samples
Epoch 1/2
100000/100000 [==============================] - 2s 20us/sample - loss: 0.5872 - dense_loss: 0.3384 - dense_1_loss: 0.2488
Epoch 2/2
100000/100000 [==============================] - 2s 17us/sample - loss: 3.6310e-05 - dense_loss: 1.6687e-05 - dense_1_loss: 1.9623e-05
dense/kernel:0: [[0.9995292]]
dense/bias:0: [-0.00030103]
dense_1/kernel:0: [[0.9994892]]
dense_1/bias:0: [-0.00027911]
dense/kernel:0
```

While fitting, we get information about both losses, since we have two of them. As expected, the weights now look much more like what we wanted. The constraint on the first dense layer forced it to train towards weight 1, and the second layer adapted correspondingly. I hope this exemplifies the power of having more control over what we put into our model and what we can receive.

Alternatively, for the importance of multiple inputs, imagine a model that takes a picture and a sentence as inputs and outputs a small story combining the two. Using multiple inputs, we can use CNN layers for the image while embedding the text separately and passing it through a few RNN layers. The outputs can then be combined for a common result.

#### Intro to GradientTape

The functional model offers a good amount of flexibility, enough for most use cases. I still think it's useful to know how to manually create and train a model. Among other benefits, it can allow for better debugging and organisation of your code. I'll be writing a post about [connectionist temporal classification](https://dl.acm.org/doi/abs/10.1145/1143844.1143891) soon, in which I expect to use this method.

We use GradientTape to create a context in which all operations on tensors are recorded. We register all operations that were performed on the input. In practice, this means that it calculates the partial derivatives of the trainable variables with respect to the loss. We correspondingly adjust the weights based on said partial derivatives. It's much easier than it sounds, let's see it in action.

```
import tensorflow as tf

dense_1 = tf.keras.layers.Dense(1)
dense_2 = tf.keras.layers.Dense(1)

# our data
data = tf.random.uniform((32*1000,1))

# define epochs, batch size
EPOCHS = 2
BATCH_SIZE = 32
LR = 0.01

# we need to handle batches manually
data = tf.reshape(data, (-1, BATCH_SIZE, 1))

# iterate through epochs
for e in range(EPOCHS):
    print(f"Epoch {e+1}")
    # every batch individually
    for batch_index, batch in enumerate(data):
        with tf.GradientTape() as g:
            # use the model to perform the prediction
            inner = dense_1(batch)
            output = dense_2(inner)
            # compute the loss
            losses = tf.keras.losses.MSE(output, batch)
            loss = tf.reduce_mean(losses)

        # retrieve the gradients of the trainable variables wrt the loss
        gradients = g.gradient(loss, [dense_1.variables, dense_2.variables])

        # print what's happening
        print(f"Batch {batch_index}/{len(data)}, Loss {loss.numpy()}"+" "*10, end="\r")

        for layer, lgrad in zip([dense_1.variables, dense_2.variables], gradients):
            for var, grad in zip(layer, lgrad):
                # apply the gradient
                var.assign_add(grad*-LR)
    print()

```

```raw
Epoch 1
Batch 999/1000, Loss 0.0004924628883600235
Epoch 2
Batch 999/1000, Loss 1.6632682218187256e-06
```

For this example, I have deliberately made life difficult for myself by defining and individually managing layers. This meant more work getting their gradients and applying them. In practice, one would normally create a model and use an optimiser to apply the gradients. Here's the above code reworked:

```
import tensorflow as tf

# define our model, no input needed
class MyModel(tf.keras.Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.dense_1 = tf.keras.layers.Dense(1)
        self.dense_2 = tf.keras.layers.Dense(1)

    def call(self, x):
        inner = self.dense_1(x)
        output = self.dense_2(inner)
        return output

# initialise our model
model = MyModel()

# our data
data = tf.random.uniform((32*1000,1))

# define epochs, batch size
EPOCHS = 2
BATCH_SIZE = 32
LR = 0.001

# we need to handle batches manually
data = tf.reshape(data, (-1, BATCH_SIZE, 1))

# easiest way to apply gradients is using an optimiser
adam = tf.keras.optimizers.Adam(learning_rate=LR)

# iterate through epochs
for e in range(EPOCHS):
    print(f"Epoch {e+1}")
    # every batch individually
    for batch_index, batch in enumerate(data):
        with tf.GradientTape() as g:
            # use the model to perform the prediction
            output = model(batch)
            # compute the loss
            losses = tf.keras.losses.MSE(output, batch)
            loss = tf.reduce_mean(losses)

        # retrieve the gradients of the trainable variables wrt the loss
        gradients = g.gradient(loss, model.trainable_variables)

        # print what's happening
        print(f"Batch {batch_index}/{len(data)}, Loss {loss.numpy()}"+" "*10, end="\r")

#         use adam to apply the gradients
        adam.apply_gradients(zip(gradients, model.variables))
    print()

```

```raw
Epoch 1
Batch 999/1000, Loss 6.90423917149019e-07
Epoch 2
Batch 999/1000, Loss 4.2168525615782215e-13
```

Doing things this way offers the most flexibility, but is much more prone to errors. Exploring it in depth would be a post by itself, which I hope to write soon. However, I hope it was useful to see the many different ways we can approach tasks in tensorflow.

Thank you for reading! I hope this post gave you some insight into the different approaches to creating our models in Tensorflow. Different tasks have different levels of complexity, and it is important to know what to pick for the job.


Creating a custom training loop in tensorflow


Keras offers a large amount of flexibility to its dense layers. In this more advanced post, I'll illustrate this and rationalise why it happens. If you are unfamiliar with shapes or the way Keras handles them, my posts [Shapes - an introduction](/posts/2020-04-07-shapes-an-introduction/) and [A closer look at shapes](/posts/2020-04-12-a-closer-look-at-shapes/) might help.

Let's dive right in with an example.

```python
import tensorflow as tf

# initialise 4 input tensors
# the tensors only contain ones
a = tf.ones((1, 2))
b = tf.ones((1, 2, 3))
c = tf.ones((1, 2, 3, 4))
d = tf.ones((1, 2, 3, 4, 5))

# add the inputs to a list
inputs = [a,b,c,d]

# make a different dense layer for each input
# since they have different shapes
dense_layers = [tf.keras.layers.Dense(2) for _ in inputs]

# run the input through its dense layer and print the shape
for i,layer in zip(inputs, dense_layers):
    print(f"Using input of shape {i.shape}")
    prediction = layer(i)
    print(f"Output shape {prediction.shape}")
    weights, bias = layer.get_weights()
    print(f"The weights shape is {weights.shape}")

    print("---")
```

```raw
Using input of shape (1, 2)
Output shape (1, 2)
The weights shape is (2, 2)
---
Using input of shape (1, 2, 3)
Output shape (1, 2, 2)
The weights shape is (3, 2)
---
Using input of shape (1, 2, 3, 4)
Output shape (1, 2, 3, 2)
The weights shape is (4, 2)
---
Using input of shape (1, 2, 3, 4, 5)
Output shape (1, 2, 3, 4, 2)
The weights shape is (5, 2)
---
```

In this example, we instantiate four input tensors of varying grade. The minimum grade allowed is two, since Keras layers always expect the first dimension to be the batch. Additionally, grade 5 is the max grade it accepts.

The printed output shows that Keras allows us to apply a dense layer on inputs with varying grades. Whichever input we give it, it gets applied over the last dimension, changing it to represent the output of the layer. Printing the weights shapes helps further illustrate the point. Furthermore, since we initialised the inputs as ones, let's have a look at the output of the third input of shape (1,2,3,4) so we can see how the weights are repeatedly applied.

```raw
tf.Tensor(
[[[[0.63791275 0.08311558]
   [0.63791275 0.08311558]
   [0.63791275 0.08311558]]

  [[0.63791275 0.08311558]
   [0.63791275 0.08311558]
   [0.63791275 0.08311558]]]], shape=(1, 2, 3, 2), dtype=float32)
```

#### It's not always this easy

This might seem perfectly expected, but Keras is usually much less tolerant. Have a look at the following example, which uses recurrent layers.

```python
import tensorflow as tf

a = tf.ones((1, 2))
b = tf.ones((1, 2, 3))
c = tf.ones((1, 2, 3, 4))

inputs = [a,b,c]

rec_layers = [tf.keras.layers.SimpleRNN(2) for _ in inputs]

# run the input through its dense layer and print the shape
for i,layer in zip(inputs, rec_layers):
    print(f"Using input of shape {i.shape} ", end="")
    try:
        prediction = layer(i)
        print(f"worked! Output shape is {prediction.shape}")
    except Exception as err:
        print(f"didn't work!")
        print(err)
    print("---")
```

```raw
Using input of shape (1, 2) didn't work!
Input 0 of layer simple_rnn_9 is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: [1, 2]
---
Using input of shape (1, 2, 3) worked! Output shape is (1, 2)
---
Using input of shape (1, 2, 3, 4) didn't work!
Input 0 of layer simple_rnn_11 is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: [1, 2, 3, 4]
---
```

Only the input with grade 3 worked! Keras interprets an input of shape (a,b,c) as meaning a batch of a items, each with b time steps, and each time step of c dimensions.

We can then even do the following.

```python
import tensorflow as tf

# Same as the first example, but
# the final dimension is kept 2
a = tf.ones((1, 2))
b = tf.ones((1, 2, 2))
c = tf.ones((1, 2, 3, 2))
d = tf.ones((1, 2, 3, 4, 2))

inputs = [a,b,c,d]

# The dense layer, only one is needed
d = tf.keras.layers.Dense(2)

# run the input through its dense layer and print the shape
for i in inputs:
    print(f"Using input of shape {i.shape}")
    prediction = d(i)
    print(f"Output shape {prediction.shape}")
    weights, bias = d.get_weights()
    print(f"The weights shape is {weights.shape}")

    print("---")
```

```raw
Using input of shape (1, 2)
Output shape (1, 2)
The weights shape is (2, 2)
---
Using input of shape (1, 2, 2)
Output shape (1, 2, 2)
The weights shape is (2, 2)
---
Using input of shape (1, 2, 3, 2)
Output shape (1, 2, 3, 2)
The weights shape is (2, 2)
---
Using input of shape (1, 2, 3, 4, 2)
Output shape (1, 2, 3, 4, 2)
The weights shape is (2, 2)
---
```

This illustrates the versatility of the dense layer, it can be applied on any tensor as long as its grade is between 2 and 5, as long as the final dimension stays the same.

#### Why is it like this?

Why, though, are we allowed to do this? There are many situations in which we want to pass a higher grade tensor into a dense layer. A simple example would be the final layer of a CTC model. Such a model ends with a bidirectional recurrent layer which returns a sequence with one item per time step. Each time step gets passed through a dense layer for the desired softmax output. The flexibility of the dense layers is helpful.

```
import tensorflow as tf

# our input of 10 timesteps
i = tf.ones((1,10,2))

# two rnn layers which return sequences, the second of which goes backwards
rnn_f = tf.keras.layers.SimpleRNN(10, return_sequences=True)
rnn_b = tf.keras.layers.SimpleRNN(10, return_sequences=True, go_backwards=True)

rnn_f_out = rnn_f(i)
rnn_b_out = rnn_b(i)

# the shapes are the same
assert rnn_f_out.shape == rnn_b_out.shape
print(f"Both outputs have the shape {rnn_f_out.shape}")

# add them together for the output of the bidirectional layer
rnn_output = rnn_f(i) + rnn_b(i)
assert rnn_output.shape == rnn_b_out.shape
print(f"Output shape of the bidirectional rnn layer is still {rnn_output.shape}")

# add a dense layer for our desired softmax output
# in this case alphabet = 4 + dash
final_dense = tf.keras.layers.Dense(5, activation="softmax")
model_output = final_dense(rnn_output)
print(f"The output shape is {model_output.shape}")
```

```raw
Both outputs have the shape (1, 10, 10)
Output shape of the bidirectional rnn layer is still (1, 10, 10)
The output shape is (1, 10, 5)
```

Dense layers being so versatile is obviously a requirement since they're such an essential element in deep neural networks and all sorts of different architectures. Without this design choice, we couldn't both create a feed-forward neural network and a CTC model. [Transformers](https://arxiv.org/abs/1706.03762) operate on tensors of shape (batch_size, num_heads, seq_len, depth) during dot product attention calculation, but they don't apply a dense layer on them. Please let me know if you know any models that pass tensors of grade 4 or 5 through a dense layer!

Thanks for reading! I hope this post helped expand on my previous shape-related posts and gives a bit of intuition into the flexibility we have when using dense layers in Keras, or at least some novel ways of playing around with data and experimenting to learn more.


on the flexibility of dense layers in Keras


Understanding shapes is an integral part of demystifying how data gets passed around in your model. If you are unfamiliar with shapes, reading my previous post, [Shapes - an introduction](/posts/2020-04-07-shapes-an-introduction/), might help. In this post, I'll explore how Tensorflow expects to receive data, and hopefully provide a better intuition for how to conceptualise your data pipeline.

We usually encounter shapes when passing tensors around. All of us have been confused by shape-related complaints coming from our ML framework of choice when first building our own datasets.

Let's dive directly into a common situation: we're starting with Tensorflow, trying to train a simple perceptron to predict 1 when we give it 1 as input:

```
import tensorflow as tf

# define our input and output
# when asked to predict 1, we train it to output 1
x = 1
y = 1

# create model, we use the keras sequential model
model = tf.keras.Sequential()

# a perceptron in keras is a Dense layer with 1 neuron
perceptron = tf.keras.layers.Dense(1)
# add it to the model
model.add(perceptron)

# compile the model to use mean squared error as the loss function
model.compile(loss="mse")

# finally, train it
model.fit(x,y)
```

```
IndexError: tuple index out of range
```

This fails. The error message is cryptic, but the answer is simple: Tensorflow always expects a batch whenever it does anything. The scalar we provided has no shape at all. The solution is to transform it into a batch by adding the number to a list. A batch of one number will have the shape (1,). This looks funny since it has a comma but only one number, but it's because a tuple cannot be defined using a single number otherwise. Here's the fix to get the previous code training:

```
x = [1]
y = [1]
```

```
Train on 1 samples
1/1 [==============================] - 0s 115ms/sample - loss: 3.8462
```

Now we're telling Tensorflow to fit on a batch of one training example and the corresponding one label. This, however, is a nicety of Tensorflow. It's kind enough to let us ignore the proper shape that we should be using. Let's expand the example to a more illustrative case in which we have multiple items in our dataset:

```
x = [-1,-1,1,1]
y = [0,0,1,1]
```

This is still fine and Tensorflow does not complain. In practice, however, we very rarely have single-dimensional data as inputs.

<figure>

![Decision surface illustration for 1- and 2-D inputs](/images/DecisionSurface.png)

<figcaption>

Usually we have multiple inputs that affect the decision surface

</figcaption>

</figure>

The problem is obvious now. We clearly need an input of shape (Batch_Size, 2) if each input has two dimensions.

```
x=[[-1,0],[0,-1],[0,1],[1,0]]
y=[0, 0, 1, 1]
```

It's, therefore, a good idea to always keep the proper shapes, if not for Tensorflow's sake, for us to better remember what role each dimension plays. The following x of shape (4,1) still runs:

```python
x = [[1],[2],[3],[4]]
```

### RNNs and CNNs

This is where many, including myself, have first encountered the shape problem. Let's first focus on RNNs, then I'll post the CNN code later. The problems and solutions are identical.

#### RNNs

Let's define a simple RNN model that takes 4 time steps of one value each and predicts a scalar.

```python
import tensorflow as tf

x=[
    [1,2,3,4],
    [4,3,2,1]
]
y=[0, 1]

model = tf.keras.Sequential()
rnn = tf.keras.layers.LSTM(1)
model.add(rnn)
perceptron = tf.keras.layers.Dense(1)
model.add(perceptron)
model.compile(loss="mse")

model.fit(x,y)
```

```
ValueError: Input 0 of layer sequential is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: [None, 4]
```

We finally get Tensorflow to complain. If this were using the same logic as the previous Dense-only code, it would happily understand that we're trying to get it to take a sequence of four numbers as the recurrent time steps. Alas, we need to be more precise. Let's reshape the input:

```python
x = tf.reshape(x, (-1, 4, 1)) # tf.Tensor([[[1][2][3][4]][[4][3][2][1]]], shape=(2, 4, 1), dtype=int32)
```

```
ValueError: Failed to find data adapter that can handle input: <class 'tensorflow.python.framework.ops.EagerTensor'>, (<class 'list'> containing values of types {"<class 'int'>"})
```

Another cryptic error message is thrown at us. It's telling us that it can't handle a list input of type int. Eagle-eyed readers would have noticed the reshape operation having output a tensor of dtype int32. However, this error message actually complains about y! We fix this error by converting y to a tensor beforehand, a good general practice to follow:

```python
y=tf.constant([0, 1])
```

```
TypeError: Input 'b' of 'MatMul' Op has type float32 that does not match type int32 of argument 'a'.
```

A final error! There's a matrix multiplication operation that received int values, even though it expected a float. This could theoretically come from the loss function, but it's more likely it's from the feed-forward pass. We already saw that x has type int32, so let's try changing that to float. We use the tensorflow _cast_ function. This runs, so here's the entire code:

```python
import tensorflow as tf

x=[
    [1,2,3,4],
    [4,3,2,1]
]
y=tf.constant([0, 1])

x = tf.reshape(x, (-1, 4, 1))
x = tf.cast(x, tf.float32)

model = tf.keras.Sequential()
rnn = tf.keras.layers.LSTM(1)
model.add(rnn)
perceptron = tf.keras.layers.Dense(1)
model.add(perceptron)
model.compile(loss="mse")

model.fit(x,y)
```

```
Train on 2 samples
2/2 [==============================] - 1s 370ms/sample - loss: 0.7737
```

#### CNNs

Here's a simple convolutional model facing the same problem as the recurrent example previously shown. Fixing it is left as an exercise to the reader.

```python
import tensorflow as tf

x=[
    [1,2,3,4],
    [4,3,2,1]
]
y=[0, 1]

model = tf.keras.Sequential()
conv = tf.keras.layers.Conv1D(1,1)
model.add(conv)
model.add(tf.keras.layers.Flatten())
perceptron = tf.keras.layers.Dense(1)
model.add(perceptron)
model.compile(loss="mse")

model.fit(x,y)
```

### Summary

I hope this post has managed to clear up some confusion regarding input shapes in Tensorflow, which often cause trouble to newcomers. Thanks for reading!


A closer look at shapes


We've all had Tensorflow complain about some shapes at some point when starting out. In this first post, I'll make shapes more intuitive by going into what shapes are, which data structures have a shape, and put everything into words.

The shape of an n-dimensional array shows its dimensions. A scalar has shape () since it has no dimensions. An array of length n has shape (n), but actually in practice (n,). This is only due to [Python tuple syntax](https://wiki.python.org/moin/TupleSyntax). A matrix would then have the shape (m,n), a cube (l,m,n) and so on.

In other words, let's take an n-dimensional array of shape (2,3,4). We can read this as follows: we have two matrices, each of which has three rows, and each of the rows has length 4.

Us humans have trouble thinking in more than three dimensions, but we can still try to make sense of n-dimensions:

- (1080,1920) could be a grey-scale full-HD picture, since we have 1080\*1920 pixels with the luminosity value
- (1080,1920,3) could be an RGB full-HD picture. Each pixel has 3 values for the three primary colours
- (900,1080,1920,3) could be 30\*30 frames = 30 seconds of full-HD video @ 30fps, a [youtubehaiku](https://www.reddit.com/r/youtubehaiku/)
- (1000,900,1080,1920,3) could be any of our embarrassingly large meme-video folders

### Python & Tensorflow

A standard Python list has no shape, since it is not enforced to be rectangular. Take the following Python structure:

```
mylist = [
    [1,2,3,4,5],
    [1,2,3,4,5,6]
]
```

The shape of _mylist_ is then ambiguous; it can either be (2,5) or (2,6). Data structures that have a shape, such as NumPy ndarrays, or TensorFlow tensors, enforce rectangularity.

```python
import tensorflow as tf

print(tf.constant(1).shape) # ()
print(tf.constant([1]).shape) # (1,)
print(tf.constant([1,2,3]).shape) # (3,)
print(tf.constant([[1],[2],[3]]).shape) # (3,1)
print(tf.constant([[1,2],[3,4],[5,6]]).shape) # (3,2)
print(tf.constant([[[1,2],[3,4],[5,6]]]).shape) # (1,3,2)
print(tf.constant([[[[1,2],[3,4],[5,6]]]]).shape) # (1,1,3,2)
```

### reshaping

Most frameworks offer useful tools to easily reshape our n-dimensional arrays or tensors. In the following snippet, I declare a tensor equal to \[1,2,3,4\] and reshape it in two different ways:

```python
import tensorflow as tf

c = tf.constant([1,2,3,4])
print(c) # tf.Tensor([1 2 3 4], shape=(4,), dtype=int32)
print(tf.reshape(c, (1,4))) # tf.Tensor([[1 2 3 4]], shape=(1, 4), dtype=int32)
print(tf.reshape(c, (4,1))) # tf.Tensor([[1][2][3][4]], shape=(4, 1), dtype=int32)
```

Sometimes we are reshaping a tensor without knowing all of its dimensions. We can still perform a reshape using -1 as one of the shapes. In this case, using -1 as one of the dimensions tells Tensorflow to figure that one out for itself based on the other numbers. At most one dimension can be set as -1, since the expression would otherwise be ambiguous.

```python
import tensorflow as tf

c = tf.constant([1,2,3,4])
print(c) # tf.Tensor([1 2 3 4], shape=(4,), dtype=int32)
print(tf.reshape(c, (-1,2))) # tf.Tensor([[1 2][3 4]], shape=(2, 2), dtype=int32)
print(tf.reshape(c, (-1,1,1,1,1))) # tf.Tensor([[[[[1]]]] [[[[2]]]] [[[[3]]]] [[[[4]]]]], shape=(4, 1, 1, 1, 1), dtype=int32)
```

Thanks for reading! I hope this post helped make shapes more intuitive if you were confused about them, as many are when starting out. If you're curious to dive further in, my next post, [A Closer Look At Shapes](/posts/2020-04-12-a-closer-look-at-shapes), has a more investigative approach into how shapes are used by Tensorflow.


What's cool to me: