3. The traceTorch Layers
It is time to (briefly) take a look at the base layer types that traceTorch presents. The practical application of the layers and when the right one should be used is discussed in more detail in the tutorials section, here we look at the toggleable features available.
traceTorch follows a slightly unconventional, but incredibly consistent and self-explanatory naming schema, encapsulating a vast
array of various neuron mechanics. All layers sit in tracetorch.snn. Detailed explanations of each layer can be found in the
references section.
LIbase name stands forLeaky Integrator, the simplest of layer types with just one trace: the membrane potential which is the direct output, no firing; commonly known asReadout, although it’s not recommended to literally make it the last layer. Internally, it holdsmemas a hidden state and sigmoid-boundbetaas a decay. At each timestep, it does the following update:mem = mem * beta + mem_delta.memdelta can be anything, such as the previous layer’s output, the synapse’s output, the recurrent output; doesn’t matter. If no thresholds are present,memis made into an Exponential Moving Average (EMA) by scaling downmemdelta bybeta:mem_delta = mem_delta * (1 - beta)~Bsuffix stands forBinary, the presence of a threshold, meaning that the layer has 2 possible outputs: a 0 or 1. Ifmemsurpasses the threshold, then the output is a 1, otherwise a 0. We subtract the threshold amount from the neuron that fired to reset it back. The threshold is bound by softplus, and is positive.~Tsuffix stands forTernary, meaning that the layer has 2 thresholds: a positive and negative one, and thus 3 possible outputs: -1, 0 or 1. Both thresholds are bound by softplus; and each one accordingly adds/subtracts the threshold amount to bringmemback closer to 0 to reset it.~Ssuffix stands forScaled, meaning that the outputs are multiplicatively scaled separately based on their polarity, used to make ternary outputs truly independent of each other. This is done with the consideration that after the SNN layer, you will have annn.Linear, or something equivalent. If we don’t scale the ternary spikes, then the true output of the pair will be-w + b,b,w + b; all equally spaced apart byw, they’re not really independent.D~prefix stads forDual, meaning that all hidden states and parameters are split into a separate positive and negative version for greater expressivity and making polarity as a truly separate, alternate signal. The “true” hidden states are calculated as the sum of their negative and positive counterparts.S~prefix stands forSynaptic, meaning that before the membrane there is a separate synaptic trace smoothing out the inputs over time before they get integrated into the membrane.syndecays withalpha, and is an EMA.R~prefix stands forRecurrent, meaning that the layer records its own outputs into a separate trace and re-integrates it back into the membrane. The trace isrecwhich decays withgamma, and is multiplicatively re-integrated intomemviarec_weight.
These combinations result in the following 28 default layers, which sit in tracetorch.snn. These layers also exits
in tracetorch.snn.flex as their LeakyIntegrator counterparts, but it’s not recommended to use the snn.flex
variants; as it carries all the extra weight of all the extra options which aren’t used. snn.flex is reserved effectively entirely for LeakyIntegrator for custom experiments.
LI- Leaky IntegratorDLI- Dual Leaky IntegratorSLI- Synaptic Leaky IntegratorDSLI- Dual Synaptic Leaky IntegratorLIB- Leaky Integrate Binary fireDLIB- Dual Leaky Integrate Binary fireSLIB- Synaptic Leaky Integrate Binary fireRLIB- Recurrent Leaky Integrate Binary fireDSLIB- Dual Synaptic Leaky Integrate Binary fireDRLIB- Dual Recurrent Leaky Integrate Binary fireSRLIB- Synaptic Recurrent Leaky Integrate Binary fireDSRLIB- Dual Synaptic Recurrent Leaky Integrate Binary fireLIT- Leaky Integrate Ternary fireDLIT- Dual Leaky Integrate Ternary fireSLIT- Synaptic Leaky Integrate Ternary fireRLIT- Recurrent Leaky Integrate Ternary fireDSLIT- Dual Synaptic Leaky Integrate Ternary fireDRLIT- Dual Recurrent Leaky Integrate Ternary fireSRLIT- Synaptic Recurrent Leaky Integrate Ternary fireDSRLIT- Dual Synaptic Recurrent Leaky Integrate Ternary fireLITS- Leaky Integrate Ternary Scaled fireDLITS- Dual Leaky Integrate Ternary Scaled fireSLITS- Synaptic Leaky Integrate Ternary Scaled fireRLITS- Recurrent Leaky Integrate Ternary Scaled fireDSLITS- Dual Synaptic Leaky Integrate Ternary Scaled fireDRLITS- Dual Recurrent Leaky Integrate Ternary Scaled fireSRLITS- Synaptic Recurrent Leaky Integrate Ternary Scaled fireDSRLITS- Dual Synaptic Recurrent Leaky Integrate Ternary Scaled fire
When initializing any layer, you can also specify the dim dimension. It’s an index as to what dimension the layer should
focus on, which defaults to -1, easily working with MLP networks. Change it to -3 and the layer now works on the color
channel of [C, H, W] or [B, C, H, W] tensors. Change it to -4 for example, and the layer works with colored voxel tensors: [C, X, Y, Z].
Each layer also asks for num_neurons, this value should be the number of values at the desired dim dimension.
So in the dim=-3 example, it would be the number of color channels.
When initializing any layer layer, each parameter also optionally asks for the following arguments:
value: the actual value it should take (instead ofvalue, it’s just the name of the parameter, such asbetaorpos_threshold, et cetera.). Can be afloatfor automatic initialization ortorch.Tensorfor a custom tensor.*_rank: the rank of the parameter. 0 means scalar (per-layer), 1 means vector (per-neuron). Defaults to 1 (vector).*_learnable: should the parameter be learnable or static. Defaults toTrue.