Using the BrainScaleS system¶

As explained in Building models, both the experiment description and the model description for the BrainScaleS system must be written as Python scripts, using the PyNN application programming interface (API), version 0.7.

In the following, the build and work flow on UHEI BrainScaleS cluster frontend nodes is described. If BrainScaleS is accessed through the Collaboratory or the Python client, the installation can be skipped.

Setup¶

export LC_ALL=C
module load nmpm_software/current

run_nmpm_software python -c "import pyhmf" && echo ok

should print ok.

The translation from the biological neuronal network description into a hardware configuration is performed by the marocco mapping tool (for more detailed information, see Details of the software stack below).

The BrainScaleS system attempts to automatically place neurons on the wafers in an optimal way. However, it is possible to influence this placement, control it manually, and examine the resultant data structures using the Python helper module pymarocco. This enables users to go from a property in PyNN (e.g. the refractory period of a single neuron within an assembly) to the corresponding parameter on hardware. A typical use case is iterative low-level tuning of hardware parameters.

Using marocco¶

import pyhmf as pynn
from pymarocco import PyMarocco

marocco = PyMarocco()
pynn.setup(marocco=marocco)

Make sure that the call to setup() happens before creating populations, if not, the populations will not be visible to marocco.

In the following example, one neuron is placed on the wafer, however, by setting marocco.backend = PyMarocco.Without, the software stops after the map & route process (i.e. before configuring the hardware system).

import pyhmf as pynn
from pymarocco import PyMarocco

marocco = PyMarocco()
marocco.backend = PyMarocco.Without

pynn.setup(marocco=marocco)

neuron = pynn.Population(1, pynn.IF_cond_exp, {})

pynn.run(10)
pynn.end()

Note

Available marocco backends are Without, Hardware, ESS. Without has been described above. Hardware is the default and performs real experiment runs on the neuromorphic hardware system. ESS runs a simulation of the hardware: the Executable System Specification.

In the output you should see:

Populations:
        0th element:    0x1f98650       Population(IF_cond_exp, 1)

If you don’t see this output, make sure that you called pynn.setup(marocco=marocco) before the call to pynn.Population.

You will also see a lot of debugging output. To set the log level, add

import pylogging
for domain in [""]:
    pylogging.set_loglevel(pylogging.get(domain), pylogging.LogLevel.ERROR)

after the import of pymarocco.

As we did not specify on which chip the neuron should be placed, marocco decides automatically to use HICANNOnWafer(X(18), Y(7)), Wafer(0) which is in the center of the wafer.

To choose the HICANN a population is placed on, we give marocco a hint:

from pyhalco_common import X,Y
from pyhalco_hicann_v2 import HICANNOnWafer

marocco.manual_placement.on_hicann(neuron, HICANNOnWafer(X(5), Y(5)))

You can inspect the coordinates on the wafer module here.

At the end, the script is the following:

#!/usr/bin/env python

import pyhmf as pynn
from pyhalco_common import Enum, X, Y
from pyhalco_hicann_v2 import HICANNOnWafer
from pymarocco import PyMarocco, Defects
from pymarocco.results import Marocco

import pylogging
for domain in ["Calibtic", "marocco"]:
    pylogging.set_loglevel(pylogging.get(domain), pylogging.LogLevel.INFO)

def get_denmems(pop, results):
    for neuron in pop:
        for item in results.placement.find(neuron):
            for denmem in item.logical_neuron():
                yield denmem

marocco = PyMarocco()
marocco.calib_backend = PyMarocco.CalibBackend.Default
marocco.defects.backend = Defects.Backend.Without
marocco.persist = "results.xml.gz"
pynn.setup(marocco = marocco)

# place the full population to a specific HICANN
pop = pynn.Population(1, pynn.IF_cond_exp)
marocco.manual_placement.on_hicann(pop, HICANNOnWafer(X(5), Y(5)), 4)

# place only parts of a population
pop2 = pynn.Population(3, pynn.IF_cond_exp)
marocco.manual_placement.on_hicann(pynn.PopulationView(pop2, [0]), HICANNOnWafer(Enum(5)))
marocco.manual_placement.on_hicann(pynn.PopulationView(pop2, [1]), HICANNOnWafer(Enum(1)))
# the third neuron will be automatically placed

pynn.run(10)
pynn.end()

results = Marocco.from_file(marocco.persist)

for denmem in get_denmems(pop, results):
    print(denmem)

for denmem in get_denmems(pop2, results):
    print(denmem)

We also added a print out of the chosen neuron circuits:

NeuronOnWafer(NeuronOnHICANN(X(0), top), HICANNOnWafer(X(5), Y(5)))
NeuronOnWafer(NeuronOnHICANN(X(1), top), HICANNOnWafer(X(5), Y(5)))
NeuronOnWafer(NeuronOnHICANN(X(0), bottom), HICANNOnWafer(X(5), Y(5)))
NeuronOnWafer(NeuronOnHICANN(X(1), bottom), HICANNOnWafer(X(5), Y(5)))

Calibration¶

To change the calibration backend from database to XML set “calib_backend” to XML. Then the calibration is looked up in xml files named w0-h84.xml, w0-h276.xml, etc. in the directory “calib_path”.

Running pyNN scripts¶

To run on the hardware one needs to use the slurm job queue system:

srun -p experiment --wmod 33 --hicann 297 run_nmpm_software python nmpm1_single_neuron.py

nmpm1_single_neuron.py:

#!/usr/bin/env python
# -*- coding: utf-8; -*-

import os
import numpy as np
import copy

from pyhalbe import HICANN
from pyhalco_hicann_v2 import Wafer, HICANNOnWafer, SynapseDriverOnHICANN
from pyhalco_hicann_v2 import RowOnSynapseDriver, FGBlockOnHICANN
from pyhalco_common import Enum, iter_all
from pysthal.command_line_util import init_logger
import pysthal

import pyhmf as pynn
from pymarocco import PyMarocco, Defects
from pymarocco.runtime import Runtime
from pymarocco.coordinates import LogicalNeuron
from pymarocco.results import Marocco

init_logger("WARN", [
    ("guidebook", "DEBUG"),
    ("marocco", "DEBUG"),
    ("Calibtic", "DEBUG"),
    ("sthal", "INFO")
])

import pylogging
logger = pylogging.get("guidebook")

neuron_parameters = {
    'cm': 0.2,
    'v_reset': -70.,
    'v_rest': -20.,
    'v_thresh': -10,
    'e_rev_I': -100.,
    'e_rev_E': 60.,
    'tau_m': 20.,
    'tau_refrac': 0.1,
    'tau_syn_E': 5.,
    'tau_syn_I': 5.,
}

marocco = PyMarocco()
marocco.default_wafer = Wafer(int(os.environ.get("WAFER", 33)))
runtime = Runtime(marocco.default_wafer)
pynn.setup(marocco=marocco, marocco_runtime=runtime)

#  ——— set up network ——————————————————————————————————————————————————————————

pop = pynn.Population(1, pynn.IF_cond_exp, neuron_parameters)

pop.record()
pop.record_v()

hicann = HICANNOnWafer(Enum(297))
marocco.manual_placement.on_hicann(pop, hicann)

connector = pynn.AllToAllConnector(weights=1)

exc_spike_times = [
    250,
    500,
    520,
    540,
    1250,
]

inh_spike_times = [
    750,
    1000,
    1020,
    1040,
    1250,
]

duration = 1500.0

stimulus_exc = pynn.Population(1, pynn.SpikeSourceArray, {
    'spike_times': exc_spike_times})
stimulus_inh = pynn.Population(1, pynn.SpikeSourceArray, {
    'spike_times': inh_spike_times})

projections = [
    pynn.Projection(stimulus_exc, pop, connector, target='excitatory'),
    pynn.Projection(stimulus_inh, pop, connector, target='inhibitory'),
]

#  ——— run mapping —————————————————————————————————————————————————————————————

marocco.skip_mapping = False
marocco.backend = PyMarocco.Without

pynn.reset()
pynn.run(duration)

#  ——— change low-level parameters before configuring hardware —————————————————

def set_sthal_params(wafer, gmax, gmax_div):
    """
    synaptic strength:
    gmax: 0 - 1023, strongest: 1023
    gmax_div: 2 - 30, strongest: 2
    """

    # for all HICANNs in use
    for hicann in wafer.getAllocatedHicannCoordinates():

        fgs = wafer[hicann].floating_gates

        # set parameters influencing the synaptic strength
        for block in iter_all(FGBlockOnHICANN):
            fgs.setShared(block, HICANN.shared_parameter.V_gmax0, gmax)
            fgs.setShared(block, HICANN.shared_parameter.V_gmax1, gmax)
            fgs.setShared(block, HICANN.shared_parameter.V_gmax2, gmax)
            fgs.setShared(block, HICANN.shared_parameter.V_gmax3, gmax)

        for driver in iter_all(SynapseDriverOnHICANN):
            for row in iter_all(RowOnSynapseDriver):
                wafer[hicann].synapses[driver][row].set_gmax_div(HICANN.GmaxDiv(gmax_div))

        # don't change values below
        for ii in range(fgs.getNoProgrammingPasses().value()):
            cfg = fgs.getFGConfig(Enum(ii))
            cfg.fg_biasn = 0
            cfg.fg_bias = 0
            fgs.setFGConfig(Enum(ii), cfg)

        for block in iter_all(FGBlockOnHICANN):
            fgs.setShared(block, HICANN.shared_parameter.V_dllres, 275)
            fgs.setShared(block, HICANN.shared_parameter.V_ccas, 800)

# call at least once
set_sthal_params(runtime.wafer(), gmax=1023, gmax_div=2)

#  ——— configure hardware ——————————————————————————————————————————————————————

marocco.skip_mapping = True
marocco.backend = PyMarocco.Hardware

# magic number from marocco
SYNAPSE_DECODER_DISABLED_SYNAPSE = HICANN.SynapseDecoder(1)

original_decoders = {}

for digital_weight in [None, 0, 5, 10, 15]:
    logger.info("running measurement with digital weight {}".format(digital_weight))
    for proj in projections:
        proj_items = runtime.results().synapse_routing.synapses().find(proj)
        for proj_item in proj_items:
            synapse = proj_item.hardware_synapse()

            proxy = runtime.wafer()[synapse.toHICANNOnWafer()].synapses[synapse]

            # make a copy of the original decoder value
            if synapse not in original_decoders:
                original_decoders[synapse] = copy.copy(proxy.decoder)

            if digital_weight != None:
                proxy.weight = HICANN.SynapseWeight(digital_weight)
                proxy.decoder = original_decoders[synapse]
            else:
                proxy.weight = HICANN.SynapseWeight(0)
                # set it to the special value that is never used for incoming addresses
                proxy.decoder = SYNAPSE_DECODER_DISABLED_SYNAPSE

    pynn.run(duration)
    np.savetxt("membrane_w{}.txt".format(digital_weight if digital_weight != None else "disabled"), pop.get_v())
    np.savetxt("spikes_w{}.txt".format(digital_weight if digital_weight != None else "disabled"), pop.getSpikes())
    pynn.reset()

    # skip checks
    marocco.verification = PyMarocco.Skip
    marocco.checkl1locking = PyMarocco.SkipCheck

# store the last result for visualization
runtime.results().save("results.xml.gz", True)

Currently, the calibration is optimized towards the neuron parameters of the example. Also note that current parameters, i.e. i_offset are not supported.

With the help of plot_spikes.py, the recorded spikes (spikes_w15.txt) and membrane trace (membrane_w15.txt) for the digital weight setting 15 can be plotted.

Membrane trace and spikes for digital weight 15

The following example shows how to sweep spike times without re-running the mapping.

nmpm1_sweep_spike_times.py:

#!/usr/bin/env python
# -*- coding: utf-8; -*-

import os
import numpy as np

from pyhalbe import HICANN
from pyhalco_hicann_v2 import Wafer, HICANNOnWafer, SynapseDriverOnHICANN
from pyhalco_hicann_v2 import RowOnSynapseDriver, FGBlockOnHICANN
from pyhalco_common import Enum, iter_all
from pysthal.command_line_util import init_logger
import pysthal

import pyhmf as pynn
from pymarocco import PyMarocco, Defects
from pymarocco.runtime import Runtime
from pymarocco.coordinates import LogicalNeuron
from pymarocco.results import Marocco

init_logger("WARN", [
    ("guidebook", "DEBUG"),
    ("marocco", "DEBUG"),
    ("Calibtic", "DEBUG"),
    ("sthal", "INFO")
])

import pylogging
logger = pylogging.get("guidebook")

neuron_parameters = {
    'cm': 0.2,
    'v_reset': -70.,
    'v_rest': -20.,
    'v_thresh': -10,
    'e_rev_I': -100.,
    'e_rev_E': 60.,
    'tau_m': 20.,
    'tau_refrac': 0.1,
    'tau_syn_E': 5.,
    'tau_syn_I': 5.,
}

marocco = PyMarocco()
marocco.default_wafer = Wafer(int(os.environ.get("WAFER", 33)))
runtime = Runtime(marocco.default_wafer)
pynn.setup(marocco=marocco, marocco_runtime=runtime)

#  ——— set up network ——————————————————————————————————————————————————————————

pop = pynn.Population(1, pynn.IF_cond_exp, neuron_parameters)

pop.record()
pop.record_v()

hicann = HICANNOnWafer(Enum(297))
marocco.manual_placement.on_hicann(pop, hicann)

connector = pynn.AllToAllConnector(weights=1)

duration = 1500.0

# initialize without spike times
stimulus_exc = pynn.Population(1, pynn.SpikeSourceArray, {'spike_times': []})
stimulus_neuron = stimulus_exc[0]

projections = [
    pynn.Projection(stimulus_exc, pop, connector, target='excitatory'),
]

#  ——— run mapping —————————————————————————————————————————————————————————————

marocco.skip_mapping = False
marocco.backend = PyMarocco.Without

pynn.reset()
pynn.run(duration)

#  ——— change low-level parameters before configuring hardware —————————————————

def set_sthal_params(wafer, gmax, gmax_div):
    """
    synaptic strength:
    gmax: 0 - 1023, strongest: 1023
    gmax_div: 2 - 30, strongest: 2
    """

    # for all HICANNs in use
    for hicann in wafer.getAllocatedHicannCoordinates():

        fgs = wafer[hicann].floating_gates

        # set parameters influencing the synaptic strength
        for block in iter_all(FGBlockOnHICANN):
            fgs.setShared(block, HICANN.shared_parameter.V_gmax0, gmax)
            fgs.setShared(block, HICANN.shared_parameter.V_gmax1, gmax)
            fgs.setShared(block, HICANN.shared_parameter.V_gmax2, gmax)
            fgs.setShared(block, HICANN.shared_parameter.V_gmax3, gmax)

        for driver in iter_all(SynapseDriverOnHICANN):
            for row in iter_all(RowOnSynapseDriver):
                wafer[hicann].synapses[driver][row].set_gmax_div(HICANN.GmaxDiv(gmax_div))

        # don't change values below
        for ii in range(fgs.getNoProgrammingPasses().value()):
            cfg = fgs.getFGConfig(Enum(ii))
            cfg.fg_biasn = 0
            cfg.fg_bias = 0
            fgs.setFGConfig(Enum(ii), cfg)

        for block in iter_all(FGBlockOnHICANN):
            fgs.setShared(block, HICANN.shared_parameter.V_dllres, 275)
            fgs.setShared(block, HICANN.shared_parameter.V_ccas, 800)

# call at least once
set_sthal_params(runtime.wafer(), gmax=1023, gmax_div=2)

#  ——— configure hardware ——————————————————————————————————————————————————————

for proj in projections:
    proj_items = runtime.results().synapse_routing.synapses().find(proj)
    for proj_item in proj_items:
        synapse = proj_item.hardware_synapse()
        proxy = runtime.wafer()[synapse.toHICANNOnWafer()].synapses[synapse]
        proxy.weight = HICANN.SynapseWeight(15)

marocco.skip_mapping = True
marocco.backend = PyMarocco.Hardware

for n, spike_times in enumerate([[100,110], [200,210], [300,310]]):

    runtime.results().spike_times.set(stimulus_neuron, spike_times)

    pynn.run(duration)
    np.savetxt("membrane_n{}.txt".format(n), pop.get_v())
    np.savetxt("spikes_n{}.txt".format(n), pop.getSpikes())
    pynn.reset()

    # skip checks
    marocco.verification = PyMarocco.Skip
    marocco.checkl1locking = PyMarocco.SkipCheck

# store the last result for visualization
runtime.results().save("results.xml.gz", True)

Inspect the synapse loss¶

When mapping network models to the wafer-scale hardware, it may happen that not all model synapses can be realized on the hardware due to limited hardware resources. Below is a simple network that is mapped to very limited resources so that synapse loss is enforced. For this example we show how to extract overall mapping statistics and projection-wise or synapse-wise synapse losses.

def main():
    """
    create small network with synapse loss.  The synapse loss happens due to a
    maximum syndriver chain length of 5 and only 4 denmems per neuron.  After
    mapping, the synapse loss per projection is evaluated and plotted for one
    projection.  The sum of lost synapses per projection is compared to the
    overall synapse loss returnd by the mapping stats.
    """
    marocco = PyMarocco()
    marocco.neuron_placement.default_neuron_size(4)
    marocco.synapse_routing.driver_chain_length(5)
    marocco.continue_despite_synapse_loss = True
    marocco.calib_backend = PyMarocco.CalibBackend.Default

    pynn.setup(marocco=marocco)

    neuron = pynn.Population(50, pynn.IF_cond_exp)
    source = pynn.Population(50, pynn.SpikeSourcePoisson, {'rate' : 2})

    connector = pynn.FixedProbabilityConnector(
            allow_self_connections=True,
            p_connect=0.5,
            weights=0.00425)
    proj_stim = pynn.Projection(source, neuron, connector, target="excitatory")
    proj_rec = pynn.Projection(neuron, neuron, connector, target="excitatory")

    pynn.run(1)

    print(marocco.stats)

    total_syns = 0
    lost_syns = 0
    for proj in [proj_stim, proj_rec]:
        l,t = projectionwise_synapse_loss(proj, marocco)
        total_syns += t
        lost_syns += l

    assert total_syns == marocco.stats.getSynapses()
    assert lost_syns == marocco.stats.getSynapseLoss()

    plot_projectionwise_synapse_loss(proj_stim, marocco)
    pynn.end()

Where print(marocco.stats) prints out overall synapse loss statistics:

MappingStats {
        synapse_loss: 581 (23.3709%)
        synapses: 2486
        synapses set: 1905
        synapses lost: 581
        synapses lost(l1): 0
        populations: 2
        projections: 2
        neurons: 50}

Invidual mapping statistics like the number of synapses set can also be directly accessed in python, see class MappingStats in the marocco documentation.

The function projectionwise_synapse_loss shows how to calculate the synapse loss per projection.

def projectionwise_synapse_loss(proj, marocco):
    """
    computes the synapse loss of a projection
    params:
      proj    - a pyhmf.Projection
      marocco -  the PyMarocco object after the mapping has run.

    returns: (nr of lost synapses, total synapses in projection)
    """
    orig_weights = proj.getWeights(format='array')
    mapped_weights = marocco.stats.getWeights(proj)
    syns = np.where(~np.isnan(orig_weights))
    realized_syns = np.where(~np.isnan(mapped_weights))
    orig = len(syns[0])
    realized = len(realized_syns[0])
    print("Projection-Wise Synapse Loss", proj, (orig - realized)*100./orig)
    return orig-realized, orig

Which yields the following output for the example above:

Projection-Wise Synapse Loss Projection ( PyAssembly (50) -> PyAssembly (50)) 23.5576923077
Projection-Wise Synapse Loss Projection ( PyAssembly (50) -> PyAssembly (50)) 23.182552504

Finally, the function plot_projectionwise_synapse_loss can be used to plot the lost and realized synapses of one projection.

def plot_projectionwise_synapse_loss(proj, marocco):
    """
    plots the realized and lost synapses of a projection
    params:
      proj    - a pyhmf.Projection
      marocco -  the PyMarocco object after the mapping has run.
    """
    orig_weights = proj.getWeights(format='array')
    mapped_weights = marocco.stats.getWeights(proj)
    realized_syns = np.where(np.isfinite(mapped_weights))
    lost_syns = np.logical_and(np.isfinite(orig_weights), np.isnan(mapped_weights))

    conn_matrix = np.zeros(orig_weights.shape)
    conn_matrix[realized_syns] =  1.
    conn_matrix[lost_syns] = 0.5

    import matplotlib
    matplotlib.use('Agg')
    import matplotlib.pyplot as plt
    plt.figure()
    plt.subplot(111)
    plt.imshow(conn_matrix, cmap='hot', interpolation='nearest')
    plt.xlabel("post neuron")
    plt.ylabel("pre neuron")
    plt.title("realized and lost synapses")
    plt.savefig("synapse_loss.png")

Realized and Lost Synapses of a Projection — Figure 74: Realized (black) and lost (red) synapses of the stimulus projection in the example network above.¶

Details of the software stack¶

The BrainScaleS Wafer-Scale Software Stack is shown in Figure 75.

User-provided neuronal network topologies are evaluated by our PyNN API implementation (PyHMF), which is written in C++ with a Python wrapper. The data structures (spike trains, populations, projections, cell types, meta information, etc.) are implemented in C++ (euter). This layer also provides a serialization and deserialization interface for lower software layers. In a nutshell, euter serializes the PyNN/PyHMF-based experiment description into a binary data stream and hands over to the next software layer. In the following software layers, the translation from this biological neuronal network description into a hardware configuration will be performed. A large fraction of the translation work, in particular the network graph translation, is performed by the marocco mapping tool (described in the PhD thesis of S. Jeltsch. Code documentation is provided by doxygen and available here.

The BrainScaleS System Software Stack — Figure 75: Data-flow-centric view of the user software stack of the BrainScaleS Wafer-Scale System. [taken from PhD thesis of E. Müller]¶

marocco uses calibration (calibtic) and blacklisting (redman) information to take into account circuit-specific properties and defects. This information is needed during the map & route process to homogenize the behavior of hardware neuron and synapse circuits and to exclude defective parts of the system.

The blacklisting works on the component level, e.g. denmems (the building blocks of neurons), on-chip buses, synapse drivers, etc. Denmems are blacklisted during calibration. If the calibration for any parameter could not be performed, the denmem is blacklisted. The number of available neurons, i.e. connected denmems, is not only depending on the number blacklisted denmems but also on their distribution. E.g. as an extreme example, if every second denmem would be blacklisted, neuron sizes larger than 1x2 (horizontal x vertical) are not possible.

The blacklisting for on-chip buses originates from the fact that every second horizontal and vertical bus is driven from a neighboring chip. If this neighbor chip can not be initialized, all buses from that chip must not be used.

FAQ¶

ImportError: No module named Coordinate¶

import pyhalbe.Coordinate or import Coordinate is deprecated and was removed from the software. Please use import pyhalco_hicann_v2 for the coordinate representation of hardware resources like HICANNOnWafer or SynapseOnHICANN and import pyhalco_common for common coordinates like Enum, iter_all, left or right instead. One option to adapt your code to the new coordinates would be to replace import Coordinate as C with

import pyhalco_hicann_v2 as C
from pyhalco_common import Enum, left, right
C.Enum, C.left, C.right = Enum, left, right

including all common coordinates you are using.

Rename PyMarocco.None to PyMarocco.Without¶

Due to the transition to python3 the marocco backend PyMarocco.None was renamed to PyMarocco.Without.

RuntimeError: Unexpected peer disconnection¶

You are using an old container, which is no longer compatible with the analog readout system. If you want to record membrane traces please update to a newer software version (2021-10-21 and newer).