Using the BrainScaleS system

As explained in Building models, both the experiment description and the model description for the BrainScaleS system must be written as Python scripts, using the PyNN application programming interface (API), version 0.7.

In the following, the build and work flow on UHEI BrainScaleS cluster frontend nodes is described. If BrainScaleS is accessed through the Collaboratory or the Python client, the installation can be skipped.

Setup

export LC_ALL=C
module load spack_visionary-defaults/2017-01-26_0.2.6
module load nmpm_software/2017-04-18-spack-2017-01-26-1
python -c "import pyhmf" && echo ok

should print ok.

The translation from the biological neuronal network description into a hardware configuration is performed by the marocco mapping tool (for more detailed information, see Details of the software stack below).

The BrainScaleS system attempts to automatically place neurons on the wafers in an optimal way. However, it is possible to influence this placement, control it manually, and examine the resultant data structures using the Python helper module pymarocco. This enables users to go from a property in PyNN (e.g. the refractory period of a single neuron within an assembly) to the corresponding parameter on hardware. A typical use case is iterative low-level tuning of hardware parameters.

Using marocco

import pyhmf as pynn
from pymarocco import PyMarocco

marocco = PyMarocco()
pynn.setup(marocco=marocco)

Make sure that the call to setup() happens before creating populations, if not, the populations will not be visible to marocco.

In the following example, one neuron is placed on the wafer, however, by setting marocco.backend = PyMarocco.None, the software stops after the map & route process (i.e. before configuring the hardware system).

import pyhmf as pynn
from pymarocco import PyMarocco

marocco = PyMarocco()
marocco.backend = PyMarocco.None

pynn.setup(marocco=marocco)

neuron = pynn.Population(1, pynn.IF_cond_exp, {})

pynn.run(10)
pynn.end()

Note

Available marocco backends are None, Hardware, ESS. None has been described above. Hardware is the default and performs real experiment runs on the neuromorphic hardware system. ESS runs a simulation of the hardware: the Executable System Specification.

In the output you should see:

Populations:
        0th element:    0x1f98650       Population(IF_cond_exp, 1)

If you don’t see this output, make sure that you called pynn.setup(marocco=marocco) before the call to pynn.Population.

You will also see a lot of debugging output. To set the log level, add

import pylogging
for domain in [""]:
    pylogging.set_loglevel(pylogging.get(domain), pylogging.LogLevel.ERROR)

after the import of pymarocco.

As we did not specify on which chip the neuron should be placed, marocco decides automatically to use HICANNOnWafer(X(18), Y(7)), Wafer(0) which is in the center of the wafer.

To choose the HICANN a population is placed on, we give marocco a hint:

import Coordinate as C

marocco.manual_placement.on_hicann(neuron, C.HICANNOnWafer(C.X(5), C.Y(5)))

At the end, the script is the following:

#!/usr/bin/env python

import pyhmf as pynn
import Coordinate as C
from pymarocco import PyMarocco
from pymarocco.results import Marocco

import pylogging
for domain in ["Calibtic", "marocco"]:
    pylogging.set_loglevel(pylogging.get(domain), pylogging.LogLevel.INFO)

marocco = PyMarocco()
marocco.persist = "results.bin"
pynn.setup(marocco = marocco)

pop = pynn.Population(1, pynn.IF_cond_exp)

marocco.manual_placement.on_hicann(pop, C.HICANNOnWafer(C.X(5), C.Y(5)), 4)

pynn.run(10)
pynn.end()

results = Marocco.from_file(marocco.persist)

for neuron in pop:
    for item in results.placement.find(neuron):
        for denmem in item.logical_neuron():
            print denmem

We also added a print out of the chosen neuron circuits:

NeuronOnWafer(NeuronOnHICANN(X(0), top), HICANNOnWafer(X(5), Y(5)))
NeuronOnWafer(NeuronOnHICANN(X(1), top), HICANNOnWafer(X(5), Y(5)))
NeuronOnWafer(NeuronOnHICANN(X(0), bottom), HICANNOnWafer(X(5), Y(5)))
NeuronOnWafer(NeuronOnHICANN(X(1), bottom), HICANNOnWafer(X(5), Y(5)))

Calibration

To change the calibration backend from database to XML set “calib_backend” to XML. Then the calibration is looked up in xml files named w0-h84.xml, w0-h276.xml, etc. in the directory “calib_path”.

Running pyNN scripts

To run on the hardware one needs to use the slurm job queue system:

srun -p experiment --wmod 33 --hicann 367 python nmpm1_single_neuron.py

nmpm1_single_neuron.py:

#!/usr/bin/env python
# -*- coding: utf-8; -*-

import numpy as np

from pyhalbe import HICANN
import pyhalbe.Coordinate as C
from pysthal.command_line_util import init_logger

import pyhmf as pynn
from pymarocco import PyMarocco, Defects
from pymarocco.runtime import Runtime
from pymarocco.coordinates import LogicalNeuron
from pymarocco.results import Marocco

init_logger("WARN", [
    ("guidebook", "DEBUG"),
    ("marocco", "DEBUG"),
    ("Calibtic", "DEBUG"),
    ("sthal", "INFO")
])

import pylogging
logger = pylogging.get("guidebook")

neuron_parameters = {
    'cm': 0.2,
    'v_reset': -70.,
    'v_rest': -20.,
    'v_thresh': -10,
    'e_rev_I': -100.,
    'e_rev_E': 60.,
    'tau_m': 20.,
    'tau_refrac': 0.1,
    'tau_syn_E': 5.,
    'tau_syn_I': 5.,
}

marocco = PyMarocco()
marocco.neuron_placement.default_neuron_size(4)
marocco.neuron_placement.minimize_number_of_sending_repeaters(False)
marocco.merger_routing.strategy(marocco.merger_routing.one_to_one)

marocco.bkg_gen_isi = 125
marocco.pll_freq = 125e6

marocco.backend = PyMarocco.Hardware
marocco.calib_backend = PyMarocco.XML
marocco.defects.path = marocco.calib_path = "/wang/data/calibration/ITL_2016"
marocco.defects.backend = Defects.XML
marocco.default_wafer = C.Wafer(33)
marocco.param_trafo.use_big_capacitors = True
marocco.input_placement.consider_firing_rate(True)
marocco.input_placement.bandwidth_utilization(0.8)

runtime = Runtime(marocco.default_wafer)
pynn.setup(marocco=marocco, marocco_runtime=runtime)

#  ——— set up network ——————————————————————————————————————————————————————————

pop = pynn.Population(1, pynn.IF_cond_exp, neuron_parameters)

pop.record()
pop.record_v()

hicann = C.HICANNOnWafer(C.Enum(367))
marocco.manual_placement.on_hicann(pop, hicann)

connector = pynn.AllToAllConnector(weights=1)

exc_spike_times = [
    250,
    500,
    520,
    540,
    1250,
]

inh_spike_times = [
    750,
    1000,
    1020,
    1040,
    1250,
]

duration = 1500.0

stimulus_exc = pynn.Population(1, pynn.SpikeSourceArray, {
    'spike_times': exc_spike_times})
stimulus_inh = pynn.Population(1, pynn.SpikeSourceArray, {
    'spike_times': inh_spike_times})

projections = [
    pynn.Projection(stimulus_exc, pop, connector, target='excitatory'),
    pynn.Projection(stimulus_inh, pop, connector, target='inhibitory'),
]

#  ——— run mapping —————————————————————————————————————————————————————————————

marocco.skip_mapping = False
marocco.backend = PyMarocco.None

pynn.reset()
pynn.run(duration)

#  ——— change low-level parameters before configuring hardware —————————————————

def set_sthal_params(wafer, gmax, gmax_div):
    """
    synaptic strength:
    gmax: 0 - 1023, strongest: 1023
    gmax_div: 1 - 15, strongest: 1
    """

    # for all HICANNs in use
    for hicann in wafer.getAllocatedHicannCoordinates():

        fgs = wafer[hicann].floating_gates

        # set parameters influencing the synaptic strength
        for block in C.iter_all(C.FGBlockOnHICANN):
            fgs.setShared(block, HICANN.shared_parameter.V_gmax0, gmax)
            fgs.setShared(block, HICANN.shared_parameter.V_gmax1, gmax)
            fgs.setShared(block, HICANN.shared_parameter.V_gmax2, gmax)
            fgs.setShared(block, HICANN.shared_parameter.V_gmax3, gmax)

        for driver in C.iter_all(C.SynapseDriverOnHICANN):
            for row in C.iter_all(C.RowOnSynapseDriver):
                wafer[hicann].synapses[driver][row].set_gmax_div(
                    C.left, gmax_div)
                wafer[hicann].synapses[driver][row].set_gmax_div(
                    C.right, gmax_div)

        # don't change values below
        for ii in xrange(fgs.getNoProgrammingPasses()):
            cfg = fgs.getFGConfig(C.Enum(ii))
            cfg.fg_biasn = 0
            cfg.fg_bias = 0
            fgs.setFGConfig(C.Enum(ii), cfg)

        for block in C.iter_all(C.FGBlockOnHICANN):
            fgs.setShared(block, HICANN.shared_parameter.V_dllres, 275)
            fgs.setShared(block, HICANN.shared_parameter.V_ccas, 800)

# call at least once
set_sthal_params(runtime.wafer(), gmax=1023, gmax_div=1)

#  ——— configure hardware ——————————————————————————————————————————————————————

marocco.skip_mapping = True
marocco.backend = PyMarocco.Hardware
# Full configuration during first step
marocco.hicann_configurator = PyMarocco.HICANNv4Configurator

for digital_weight in [5, 10, 15]:
    logger.info("running measurement with digital weight {}".format(digital_weight))
    for proj in projections:
        proj_item, = runtime.results().synapse_routing.synapses().find(proj)
        synapse = proj_item.hardware_synapse()

        proxy = runtime.wafer()[synapse.toHICANNOnWafer()].synapses[synapse]
        proxy.weight = HICANN.SynapseWeight(digital_weight)

    pynn.run(duration)
    np.savetxt("membrane_w{}.txt".format(digital_weight), pop.get_v())
    np.savetxt("spikes_w{}.txt".format(digital_weight), pop.getSpikes())
    pynn.reset()

    # only change digital parameters from now on
    marocco.hicann_configurator = PyMarocco.NoResetNoFGConfigurator

Currently, the calibration is optimized towards the neuron parameters of the example. Also note that current parameters, i.e. i_offset are not supported.

With the help of plot_spikes.py, the recorded spikes (spikes_w15.txt) and membrane trace (membrane_w15.txt) for the digital weight setting 15 can be plotted.

Membrane trace and spikes for digital weight 15

Inspect the synapse loss

When mapping network models to the wafer-scale hardware, it may happen that not all model synapses can be realized on the hardware due to limited hardware resources. Below is a simple network that is mapped to very limited resources so that synapse loss is enforced. For this example we show how to extract overall mapping statistics and projection-wise or synapse-wise synapse losses.

def main():
    """
    create small network with synapse loss.  The synapse loss happens due to a
    maximum syndriver chain length of 5 and only 4 denmems per neuron.  After
    mapping, the synapse loss per projection is evaluated and plotted for one
    projection.  The sum of lost synapses per projection is compared to the
    overall synapse loss returnd by the mapping stats.
    """
    marocco = PyMarocco()
    marocco.neuron_placement.default_neuron_size(4)
    marocco.synapse_routing.driver_chain_length(5)

    pynn.setup(marocco=marocco)

    neuron = pynn.Population(50, pynn.IF_cond_exp)
    source = pynn.Population(50, pynn.SpikeSourcePoisson, {'rate' : 2})

    connector = pynn.FixedProbabilityConnector(
            allow_self_connections=True,
            p_connect=0.5,
            weights=0.00425)
    proj_stim = pynn.Projection(source, neuron, connector, target="excitatory")
    proj_rec = pynn.Projection(neuron, neuron, connector, target="excitatory")

    pynn.run(1)

    print marocco.stats

    total_syns = 0
    lost_syns = 0
    for proj in [proj_stim, proj_rec]:
        l,t = projectionwise_synapse_loss(proj, marocco)
        total_syns += t
        lost_syns += l

    assert total_syns == marocco.stats.getSynapses()
    assert lost_syns == marocco.stats.getSynapseLoss()

    plot_projectionwise_synapse_loss(proj_stim, marocco)
    pynn.end()

Where print marocco.stats prints out overall synapse loss statistics:

MappingStats {
        synapse_loss: 581 (23.3709%)
        synapses: 2486
        synapses set: 1905
        synapses lost: 581
        synapses lost(l1): 0
        populations: 2
        projections: 2
        neurons: 50}

Invidual mapping statistics like the number of synapses set can also be directly accessed in python, see class MappingStats in the marocco documentation.

The function projectionwise_synapse_loss shows how to calculate the synapse loss per projection.

def projectionwise_synapse_loss(proj, marocco):
    """
    computes the synapse loss of a projection
    params:
      proj    - a pyhmf.Projection
      marocco -  the PyMarocco object after the mapping has run.

    returns: (nr of lost synapses, total synapses in projection)
    """
    orig_weights = proj.getWeights(format='array')
    mapped_weights = marocco.stats.getWeights(proj)
    syns = np.where(~np.isnan(orig_weights))
    realized_syns = np.where(~np.isnan(mapped_weights))
    orig = len(syns[0])
    realized = len(realized_syns[0])
    print "Projection-Wise Synapse Loss", proj, (orig - realized)*100./orig
    return orig-realized, orig

Which yields the following output for the example above:

Projection-Wise Synapse Loss Projection ( PyAssembly (50) -> PyAssembly (50)) 23.5576923077
Projection-Wise Synapse Loss Projection ( PyAssembly (50) -> PyAssembly (50)) 23.182552504

Finally, the function plot_projectionwise_synapse_loss can be used to plot the lost and realized synapses of one projection.

def plot_projectionwise_synapse_loss(proj, marocco):
    """
    plots the realized and lost synapses of a projection
    params:
      proj    - a pyhmf.Projection
      marocco -  the PyMarocco object after the mapping has run.
    """
    orig_weights = proj.getWeights(format='array')
    mapped_weights = marocco.stats.getWeights(proj)
    realized_syns = np.where(np.isfinite(mapped_weights))
    lost_syns = np.logical_and(np.isfinite(orig_weights), np.isnan(mapped_weights))

    conn_matrix = np.zeros(orig_weights.shape)
    conn_matrix[realized_syns] =  1.
    conn_matrix[lost_syns] = 0.5

    import matplotlib
    matplotlib.use('Agg')
    import matplotlib.pyplot as plt
    plt.figure()
    plt.subplot(111)
    plt.imshow(conn_matrix, cmap='hot', interpolation='nearest')
    plt.xlabel("post neuron")
    plt.ylabel("pre neuron")
    plt.title("realized and lost synapses")
    plt.savefig("synapse_loss.png")
Realized and Lost Synapses of a Projection

Figure 111: Realized (black) and lost (red) synapses of the stimulus projection in the example network above.

Details of the software stack

The BrainScaleS Wafer-Scale Software Stack is shown in Figure 112.

User-provided neuronal network topologies are evaluated by our PyNN API implementation (PyHMF), which is written in C++ with a Python wrapper. The data structures (spike trains, populations, projections, cell types, meta information, etc.) are implemented in C++ (euter). This layer also provides a serialization and deserialization interface for lower software layers. In a nutshell, euter serializes the PyNN/PyHMF-based experiment description into a binary data stream and hands over to the next software layer. In the following software layers, the translation from this biological neuronal network description into a hardware configuration will be performed. A large fraction of the translation work, in particular the network graph translation, is performed by the marocco mapping tool (described in the PhD thesis of S. Jeltsch. Code documentation is provided by doxygen and available here.

The BrainScaleS System Software Stack

Figure 112: Data-flow-centric view of the user software stack of the BrainScaleS Wafer-Scale System. [taken from PhD thesis of E. Müller]

marocco uses calibration (calibtic) and blacklisting (redman) information to take into account circuit-specific properties and defects. This information is needed during the map & route process to homogenize the behavior of hardware neuron and synapse circuits and to exclude defective parts of the system.