4. The Distributed Hidden State
===============================

As with any recurrent architecture, the biggest challenge is for the model to figure out what to keep and what to discard.
Confined by working memory (hidden state(s)), the model cannot simply store absolutely everything. Transformers work because
we keep everything in context, and dynamically create an analogue for the hidden state based on the current sequence length.

Perhaps the most well known of RNN architectures is the Long Short-Term Memory (LSTM). It solves the problem of figuring
out what to remember and what to forget by utilizing gates to control the flow of information. It works with a literal
``hidden`` and ``cell`` hidden states (short and long term), and uses gates that work on sigmoid (values between 0 and 1)
to control how much of the information to keep, how much to forget; how to update the state and what to return. The more
modern and simplified alternative to LSTMs is the Gated Recurrent Unit (GRU), which only use 2 gates instead of 3,
and 1 hidden state instead of 2, but the principle remains effectively identical: gates decide what's important and what
isn't.

SNNs on the other hand are simultaneously significantly simpler in structure and more complex in interpretation, working
on a much higher, abstract level than literal gates controlling the portion of information to keep.

In the simplest configuration, the ``LIB`` neuron works like any MLP, except with the option of dipping into time. ``mem``
stores the amount of information the neuron has accumulated, which gradually decays because of ``beta``. Firing means that
the neuron has accumulated enough information to flip the gate and signal to the downstream layer. A high ``beta`` (close to 1)
means that information is retained over a long period of time. The formula for approximating the number of timesteps
that information is retained for is ``1 / (1 - beta)``. If ``beta`` is 0.9, then it remembers the last 10 timesteps. If it's
0.99, then 100 timesteps, and so on. ``beta`` dictates the temporal window that the neuron remembers information over.
But even then, the moment that ``mem`` reaches the threshold, it resets, and now the neuron has lost all the information
it has accumulated. As you can see, this is quite problematic, as no matter how high a number we pick, we're always
constrained by the highest ``beta`` value in the entire network. The model fundamentally cannot remember information for
longer than that, and that's assuming the perfect configuration.

So, we add synaptic dynamics with ``syn`` and ``alpha`` decay. ``syn`` is an Exponential Moving Average (EMA) of the input
that the neuron receives at each timestep, and then passes that value in to ``mem``. If ``alpha`` is close to 0, then
it's as if it didn't exist as the signal is instantly integrated into ``syn``, the old information is lost, and thus the
only thing that gets to ``mem`` is the newly received input. However, the closer that ``alpha`` is to 1, the longer its
temporal window is. If ``alpha`` is 0.9, then that means that the value stored in ``syn`` is effectively an EMA of the
past 10 timesteps, and now, ``mem`` receives not the instantaneous information, but the information that the neuron has
received over time. Now, even if the neuron fires and ``mem`` resets, it's not as critical because ``syn`` can re-fill
it in the next timestep given the proper circumstances. Even now, this isn't enough. We still can't reach a theoretically
infinite attention window.

We can add bipolar spiking, which means having a negative threshold in conjunction with the positive threshold. This now
means that negative and positive signals are separate in nature, it's not as simple as "positive = excitatory, negative =
inhibitory". Negative and positive spikes are scaled by ``pos_scale`` and ``neg_scale``, which means that the neuron can
learn to output 3 independent signals. It can learn to never output positive or negative spikes by tuning the according
``*_scale`` down to 0. It can learn to have a static output by having both ``*_scale`` down at 0; it can even learn to
ignore polarity make positive and negative spikes be of one polarity by making one of the ``*_scale`` the negative of the
other ``*_scale``. With bipolar spiking, a true inhibitory signal is now a 0, not just negative values, and for this to
make sense and work, the magnitude of the signal becomes an even more important factor than before, and thus accordingly
the ``pos_threshold`` and ``neg_threshold`` learn independently: the closer they are to zero, the more sensitive the neuron
effectively is to the respective signal. But this is merely an expansion of the capability, we still have not yet reached
infinite context.

Having already separated positive and negative signals by separate thresholds and scales, it would also make sense that we
have separate traces for them too. Thus we introduce Duality: for each hidden state and for each parameter, we have a positive and negative variant,
and attain the "true" hidden state by summing the two. Different decays for different polarities means that we can unlock
unique dynamics: different speeds at which the neuron adapts based on the polarity of the signal it receives, such as
fast adaptation to positive signals but slow adaptation to negative signals. We can introduce this duality to *any* hidden
state, which means we quickly unlock another vast amount of dynamics which are in some way meaningful.

And finally, we add recurrence. ``rec`` stores the running average of the neuron's output, decayed via ``gamma``, and re-integrated
back into ``mem`` at each timestep, multiplicatively scaled by ``rec_weight``. It is with recurrence that we are finally
able to unlock complex firing patterns and self-supporting states. If ``rec`` is positive (the model output positive spikes),
and ``rec_weight`` is positive, then the neuron can enter a permanent firing state that requires heavy external suppression
in order to stop firing. On the contrary, if ``rec_weight`` is negative, and suppose that the ``gamma`` decay is very low
(fast reset, small temporal window), then the instant that the neuron outputs a positive spike, it will make ``rec``
positive too, but the negative ``rec_weight`` will re-integrate ``rec`` back into ``mem`` to make it negative, and thus
the model will constantly alternate between positive and negative spike outputs. Adjusting the ``beta`` and ``gamma`` decay
and we can get arbitrary spacing between the alternating patterns: [-1,1,-1,1] or [-1,0,1,0,-1], et cetera. We also add
``bias``, a value that just directly gets integrated into ``mem`` at each timestep. Considering bipolar spiking, and the
learnable ``pos_scale`` and ``neg_scale``, we can create an autotelic neuron that without any input has a certain firing
pattern, and reach virtually any firing sequence we need. They can function as conditional tempo clocks, working at different
speeds in tandem, and downstream layers base their outputs on how they align, akin to how fourier series works.

Parallelize the neurons, initialize them with different (can even do random) ``alpha``, ``beta``, ``gamma``, ``pos_threshold``,
``neg_threshold``, ``pos_scale``, ``neg_scale``, ``rec_weight``, ``bias``, consider the doubling due to duality; and the number of dynamics we begin with and can already
capture is unfathomably vast. Chain the layers, and it would be surprising if the model *couldn't* learn some function.
Use residual connections, pairing each SNN layer with an ``nn.Linear`` or equivalent, make them take as input a floating
point vector (electrical current for the neurons), and as output return the sum of the input and the output spikes passed
through the ``nn.Linear``, and you've solved the issue of vanishing gradients, and each layer is effectively functioning
as a high level, recurrent tensor editor.

Very quickly, interpretability becomes something very difficult to do. ``mem``, ``syn``, ``rec`` are simple enough to understand
in principle, but combine the three together, and consider the presence of downstream and upstream layers taking play, and
the dynamics that are now possible become far more than blind signal accumulation. We thus finally achieve the Distributed
Hidden State (DHS). Each hidden state in itself means very little, and it is when they work together that we get something
abstract but meaningful.