# Using the BrainScaleS system¶

As explained in Building models, both the experiment description and the model description for the BrainScaleS system must be written as Python scripts, using the PyNN application programming interface (API), version 0.7.

In the following, the build and work flow on UHEI BrainScaleS cluster frontend nodes is described. If BrainScaleS is accessed through the Collaboratory or the Python client, the installation can be skipped.

## Setup¶

export LC_ALL=C

python -c "import pyhmf" && echo ok


should print ok.

The translation from the biological neuronal network description into a hardware configuration is performed by the marocco mapping tool (for more detailed information, see Details of the software stack below).

The BrainScaleS system attempts to automatically place neurons on the wafers in an optimal way. However, it is possible to influence this placement, control it manually, and examine the resultant data structures using the Python helper module pymarocco. This enables users to go from a property in PyNN (e.g. the refractory period of a single neuron within an assembly) to the corresponding parameter on hardware. A typical use case is iterative low-level tuning of hardware parameters.

## Using marocco¶

import pyhmf as pynn
from pymarocco import PyMarocco

marocco = PyMarocco()
pynn.setup(marocco=marocco)


Make sure that the call to setup() happens before creating populations, if not, the populations will not be visible to marocco.

In the following example, one neuron is placed on the wafer, however, by setting marocco.backend = PyMarocco.None, the software stops after the map & route process (i.e. before configuring the hardware system).

import pyhmf as pynn
from pymarocco import PyMarocco

marocco = PyMarocco()
marocco.backend = PyMarocco.None

pynn.setup(marocco=marocco)

neuron = pynn.Population(1, pynn.IF_cond_exp, {})

pynn.run(10)
pynn.end()


Note

Available marocco backends are None, Hardware, ESS. None has been described above. Hardware is the default and performs real experiment runs on the neuromorphic hardware system. ESS runs a simulation of the hardware: the Executable System Specification.

In the output you should see:

Populations:
0th element:    0x1f98650       Population(IF_cond_exp, 1)


If you don’t see this output, make sure that you called pynn.setup(marocco=marocco) before the call to pynn.Population.

You will also see a lot of debugging output. To set the log level, add

import pylogging
for domain in [""]:
pylogging.set_loglevel(pylogging.get(domain), pylogging.LogLevel.ERROR)


after the import of pymarocco.

As we did not specify on which chip the neuron should be placed, marocco decides automatically to use HICANNOnWafer(X(18), Y(7)), Wafer(0) which is in the center of the wafer.

To choose the HICANN a population is placed on, we give marocco a hint:

import Coordinate as C

marocco.manual_placement.on_hicann(neuron, C.HICANNOnWafer(C.X(5), C.Y(5)))


At the end, the script is the following:

#!/usr/bin/env python

import pyhmf as pynn
import Coordinate as C
from pymarocco import PyMarocco, Defects
from pymarocco.results import Marocco

import pylogging
for domain in ["Calibtic", "marocco"]:
pylogging.set_loglevel(pylogging.get(domain), pylogging.LogLevel.INFO)

marocco = PyMarocco()
marocco.calib_backend = PyMarocco.CalibBackend.Default
marocco.defects.backend = Defects.Backend.None
marocco.persist = "results.bin"
pynn.setup(marocco = marocco)

pop = pynn.Population(1, pynn.IF_cond_exp)

marocco.manual_placement.on_hicann(pop, C.HICANNOnWafer(C.X(5), C.Y(5)), 4)

pynn.run(10)
pynn.end()

results = Marocco.from_file(marocco.persist)

for neuron in pop:
for item in results.placement.find(neuron):
for denmem in item.logical_neuron():
print denmem


We also added a print out of the chosen neuron circuits:

NeuronOnWafer(NeuronOnHICANN(X(0), top), HICANNOnWafer(X(5), Y(5)))
NeuronOnWafer(NeuronOnHICANN(X(1), top), HICANNOnWafer(X(5), Y(5)))
NeuronOnWafer(NeuronOnHICANN(X(0), bottom), HICANNOnWafer(X(5), Y(5)))
NeuronOnWafer(NeuronOnHICANN(X(1), bottom), HICANNOnWafer(X(5), Y(5)))


### Calibration¶

To change the calibration backend from database to XML set “calib_backend” to XML. Then the calibration is looked up in xml files named w0-h84.xml, w0-h276.xml, etc. in the directory “calib_path”.

### Running pyNN scripts¶

To run on the hardware one needs to use the slurm job queue system:

srun -p experiment --wmod 33 --hicann 297 python nmpm1_single_neuron.py


nmpm1_single_neuron.py:

#!/usr/bin/env python
# -*- coding: utf-8; -*-

import os
import numpy as np

from pyhalbe import HICANN
import pyhalbe.Coordinate as C
from pysthal.command_line_util import init_logger

import pyhmf as pynn
from pymarocco import PyMarocco, Defects
from pymarocco.runtime import Runtime
from pymarocco.coordinates import LogicalNeuron
from pymarocco.results import Marocco

init_logger("WARN", [
("guidebook", "DEBUG"),
("marocco", "DEBUG"),
("Calibtic", "DEBUG"),
("sthal", "INFO")
])

import pylogging
logger = pylogging.get("guidebook")

neuron_parameters = {
'cm': 0.2,
'v_reset': -70.,
'v_rest': -20.,
'v_thresh': -10,
'e_rev_I': -100.,
'e_rev_E': 60.,
'tau_m': 20.,
'tau_refrac': 0.1,
'tau_syn_E': 5.,
'tau_syn_I': 5.,
}

marocco = PyMarocco()
marocco.neuron_placement.default_neuron_size(4)
marocco.neuron_placement.minimize_number_of_sending_repeaters(False)
marocco.merger_routing.strategy(marocco.merger_routing.one_to_one)

marocco.bkg_gen_isi = 125
marocco.pll_freq = 125e6

marocco.backend = PyMarocco.Hardware
marocco.calib_backend = PyMarocco.XML
marocco.defects.path = marocco.calib_path = "/wang/data/calibration/brainscales/default-2017-09-26-1"
marocco.defects.backend = Defects.XML
marocco.default_wafer = C.Wafer(int(os.environ.get("WAFER", 33)))
marocco.param_trafo.use_big_capacitors = True
marocco.input_placement.consider_firing_rate(True)
marocco.input_placement.bandwidth_utilization(0.8)

runtime = Runtime(marocco.default_wafer)
pynn.setup(marocco=marocco, marocco_runtime=runtime)

#  ——— set up network ——————————————————————————————————————————————————————————

pop = pynn.Population(1, pynn.IF_cond_exp, neuron_parameters)

pop.record()
pop.record_v()

hicann = C.HICANNOnWafer(C.Enum(297))
marocco.manual_placement.on_hicann(pop, hicann)

connector = pynn.AllToAllConnector(weights=1)

exc_spike_times = [
250,
500,
520,
540,
1250,
]

inh_spike_times = [
750,
1000,
1020,
1040,
1250,
]

duration = 1500.0

stimulus_exc = pynn.Population(1, pynn.SpikeSourceArray, {
'spike_times': exc_spike_times})
stimulus_inh = pynn.Population(1, pynn.SpikeSourceArray, {
'spike_times': inh_spike_times})

projections = [
pynn.Projection(stimulus_exc, pop, connector, target='excitatory'),
pynn.Projection(stimulus_inh, pop, connector, target='inhibitory'),
]

#  ——— run mapping —————————————————————————————————————————————————————————————

marocco.skip_mapping = False
marocco.backend = PyMarocco.None

pynn.reset()
pynn.run(duration)

#  ——— change low-level parameters before configuring hardware —————————————————

def set_sthal_params(wafer, gmax, gmax_div):
"""
synaptic strength:
gmax: 0 - 1023, strongest: 1023
gmax_div: 1 - 15, strongest: 1
"""

# for all HICANNs in use
for hicann in wafer.getAllocatedHicannCoordinates():

fgs = wafer[hicann].floating_gates

# set parameters influencing the synaptic strength
for block in C.iter_all(C.FGBlockOnHICANN):
fgs.setShared(block, HICANN.shared_parameter.V_gmax0, gmax)
fgs.setShared(block, HICANN.shared_parameter.V_gmax1, gmax)
fgs.setShared(block, HICANN.shared_parameter.V_gmax2, gmax)
fgs.setShared(block, HICANN.shared_parameter.V_gmax3, gmax)

for driver in C.iter_all(C.SynapseDriverOnHICANN):
for row in C.iter_all(C.RowOnSynapseDriver):
wafer[hicann].synapses[driver][row].set_gmax_div(
C.left, gmax_div)
wafer[hicann].synapses[driver][row].set_gmax_div(
C.right, gmax_div)

# don't change values below
for ii in xrange(fgs.getNoProgrammingPasses()):
cfg = fgs.getFGConfig(C.Enum(ii))
cfg.fg_biasn = 0
cfg.fg_bias = 0
fgs.setFGConfig(C.Enum(ii), cfg)

for block in C.iter_all(C.FGBlockOnHICANN):
fgs.setShared(block, HICANN.shared_parameter.V_dllres, 275)
fgs.setShared(block, HICANN.shared_parameter.V_ccas, 800)

# call at least once
set_sthal_params(runtime.wafer(), gmax=1023, gmax_div=1)

#  ——— configure hardware ——————————————————————————————————————————————————————

marocco.skip_mapping = True
marocco.backend = PyMarocco.Hardware
# Full configuration during first step
marocco.hicann_configurator = PyMarocco.ParallelHICANNv4Configurator

for digital_weight in [5, 10, 15]:
logger.info("running measurement with digital weight {}".format(digital_weight))
for proj in projections:
proj_item, = runtime.results().synapse_routing.synapses().find(proj)
synapse = proj_item.hardware_synapse()

proxy = runtime.wafer()[synapse.toHICANNOnWafer()].synapses[synapse]
proxy.weight = HICANN.SynapseWeight(digital_weight)

pynn.run(duration)
np.savetxt("membrane_w{}.txt".format(digital_weight), pop.get_v())
np.savetxt("spikes_w{}.txt".format(digital_weight), pop.getSpikes())
pynn.reset()

# only change digital parameters from now on
marocco.hicann_configurator = PyMarocco.NoResetNoFGConfigurator


Currently, the calibration is optimized towards the neuron parameters of the example. Also note that current parameters, i.e. i_offset are not supported.

With the help of plot_spikes.py, the recorded spikes (spikes_w15.txt) and membrane trace (membrane_w15.txt) for the digital weight setting 15 can be plotted.

The following example shows how to sweep spike times without re-running the mapping.

nmpm1_sweep_spike_times.py:

#!/usr/bin/env python
# -*- coding: utf-8; -*-

import os
import numpy as np

from pyhalbe import HICANN
import pyhalbe.Coordinate as C
from pysthal.command_line_util import init_logger

import pyhmf as pynn
from pymarocco import PyMarocco, Defects
from pymarocco.runtime import Runtime
from pymarocco.coordinates import LogicalNeuron
from pymarocco.results import Marocco

init_logger("WARN", [
("guidebook", "DEBUG"),
("marocco", "DEBUG"),
("Calibtic", "DEBUG"),
("sthal", "INFO")
])

import pylogging
logger = pylogging.get("guidebook")

neuron_parameters = {
'cm': 0.2,
'v_reset': -70.,
'v_rest': -20.,
'v_thresh': -10,
'e_rev_I': -100.,
'e_rev_E': 60.,
'tau_m': 20.,
'tau_refrac': 0.1,
'tau_syn_E': 5.,
'tau_syn_I': 5.,
}

marocco = PyMarocco()
marocco.neuron_placement.default_neuron_size(4)
marocco.neuron_placement.minimize_number_of_sending_repeaters(False)
marocco.merger_routing.strategy(marocco.merger_routing.one_to_one)

marocco.bkg_gen_isi = 125
marocco.pll_freq = 125e6

marocco.backend = PyMarocco.Hardware
marocco.calib_backend = PyMarocco.XML
marocco.defects.path = marocco.calib_path = "/wang/data/calibration/brainscales/default-2017-09-26-1"
marocco.defects.backend = Defects.XML
marocco.default_wafer = C.Wafer(int(os.environ.get("WAFER", 33)))
marocco.param_trafo.use_big_capacitors = True
marocco.input_placement.consider_firing_rate(True)
marocco.input_placement.bandwidth_utilization(0.8)

runtime = Runtime(marocco.default_wafer)
pynn.setup(marocco=marocco, marocco_runtime=runtime)

#  ——— set up network ——————————————————————————————————————————————————————————

pop = pynn.Population(1, pynn.IF_cond_exp, neuron_parameters)

pop.record()
pop.record_v()

hicann = C.HICANNOnWafer(C.Enum(297))
marocco.manual_placement.on_hicann(pop, hicann)

connector = pynn.AllToAllConnector(weights=1)

duration = 1500.0

# initialize without spike times
stimulus_exc = pynn.Population(1, pynn.SpikeSourceArray, {'spike_times': []})
stimulus_neuron = stimulus_exc[0]

projections = [
pynn.Projection(stimulus_exc, pop, connector, target='excitatory'),
]

#  ——— run mapping —————————————————————————————————————————————————————————————

marocco.skip_mapping = False
marocco.backend = PyMarocco.None

pynn.reset()
pynn.run(duration)

#  ——— change low-level parameters before configuring hardware —————————————————

def set_sthal_params(wafer, gmax, gmax_div):
"""
synaptic strength:
gmax: 0 - 1023, strongest: 1023
gmax_div: 1 - 15, strongest: 1
"""

# for all HICANNs in use
for hicann in wafer.getAllocatedHicannCoordinates():

fgs = wafer[hicann].floating_gates

# set parameters influencing the synaptic strength
for block in C.iter_all(C.FGBlockOnHICANN):
fgs.setShared(block, HICANN.shared_parameter.V_gmax0, gmax)
fgs.setShared(block, HICANN.shared_parameter.V_gmax1, gmax)
fgs.setShared(block, HICANN.shared_parameter.V_gmax2, gmax)
fgs.setShared(block, HICANN.shared_parameter.V_gmax3, gmax)

for driver in C.iter_all(C.SynapseDriverOnHICANN):
for row in C.iter_all(C.RowOnSynapseDriver):
wafer[hicann].synapses[driver][row].set_gmax_div(
C.left, gmax_div)
wafer[hicann].synapses[driver][row].set_gmax_div(
C.right, gmax_div)

# don't change values below
for ii in xrange(fgs.getNoProgrammingPasses()):
cfg = fgs.getFGConfig(C.Enum(ii))
cfg.fg_biasn = 0
cfg.fg_bias = 0
fgs.setFGConfig(C.Enum(ii), cfg)

for block in C.iter_all(C.FGBlockOnHICANN):
fgs.setShared(block, HICANN.shared_parameter.V_dllres, 275)
fgs.setShared(block, HICANN.shared_parameter.V_ccas, 800)

# call at least once
set_sthal_params(runtime.wafer(), gmax=1023, gmax_div=1)

#  ——— configure hardware ——————————————————————————————————————————————————————

for proj in projections:
proj_item, = runtime.results().synapse_routing.synapses().find(proj)
synapse = proj_item.hardware_synapse()
proxy = runtime.wafer()[synapse.toHICANNOnWafer()].synapses[synapse]
proxy.weight = HICANN.SynapseWeight(15)

marocco.skip_mapping = True
marocco.backend = PyMarocco.Hardware
# Full configuration during first step
marocco.hicann_configurator = PyMarocco.ParallelHICANNv4Configurator

for n, spike_times in enumerate([[100,110], [200,210], [300,310]]):

runtime.results().spike_times.set(stimulus_neuron, spike_times)

pynn.run(duration)
np.savetxt("membrane_n{}.txt".format(n), pop.get_v())
np.savetxt("spikes_n{}.txt".format(n), pop.getSpikes())
pynn.reset()

# only change digital parameters from now on
marocco.hicann_configurator = PyMarocco.NoResetNoFGConfigurator

# skip checks
marocco.verification = PyMarocco.Skip
marocco.checkl1locking = PyMarocco.SkipCheck


## Inspect the synapse loss¶

When mapping network models to the wafer-scale hardware, it may happen that not all model synapses can be realized on the hardware due to limited hardware resources. Below is a simple network that is mapped to very limited resources so that synapse loss is enforced. For this example we show how to extract overall mapping statistics and projection-wise or synapse-wise synapse losses.

def main():
"""
create small network with synapse loss.  The synapse loss happens due to a
maximum syndriver chain length of 5 and only 4 denmems per neuron.  After
mapping, the synapse loss per projection is evaluated and plotted for one
projection.  The sum of lost synapses per projection is compared to the
overall synapse loss returnd by the mapping stats.
"""
marocco = PyMarocco()
marocco.neuron_placement.default_neuron_size(4)
marocco.synapse_routing.driver_chain_length(5)
marocco.continue_despite_synapse_loss = True

pynn.setup(marocco=marocco)

neuron = pynn.Population(50, pynn.IF_cond_exp)
source = pynn.Population(50, pynn.SpikeSourcePoisson, {'rate' : 2})

connector = pynn.FixedProbabilityConnector(
allow_self_connections=True,
p_connect=0.5,
weights=0.00425)
proj_stim = pynn.Projection(source, neuron, connector, target="excitatory")
proj_rec = pynn.Projection(neuron, neuron, connector, target="excitatory")

pynn.run(1)

print marocco.stats

total_syns = 0
lost_syns = 0
for proj in [proj_stim, proj_rec]:
l,t = projectionwise_synapse_loss(proj, marocco)
total_syns += t
lost_syns += l

assert total_syns == marocco.stats.getSynapses()
assert lost_syns == marocco.stats.getSynapseLoss()

plot_projectionwise_synapse_loss(proj_stim, marocco)
pynn.end()


Where print marocco.stats prints out overall synapse loss statistics:

MappingStats {
synapse_loss: 581 (23.3709%)
synapses: 2486
synapses set: 1905
synapses lost: 581
synapses lost(l1): 0
populations: 2
projections: 2
neurons: 50}


Invidual mapping statistics like the number of synapses set can also be directly accessed in python, see class MappingStats in the marocco documentation.

The function projectionwise_synapse_loss shows how to calculate the synapse loss per projection.

def projectionwise_synapse_loss(proj, marocco):
"""
computes the synapse loss of a projection
params:
proj    - a pyhmf.Projection
marocco -  the PyMarocco object after the mapping has run.

returns: (nr of lost synapses, total synapses in projection)
"""
orig_weights = proj.getWeights(format='array')
mapped_weights = marocco.stats.getWeights(proj)
syns = np.where(~np.isnan(orig_weights))
realized_syns = np.where(~np.isnan(mapped_weights))
orig = len(syns[0])
realized = len(realized_syns[0])
print "Projection-Wise Synapse Loss", proj, (orig - realized)*100./orig
return orig-realized, orig


Which yields the following output for the example above:

Projection-Wise Synapse Loss Projection ( PyAssembly (50) -> PyAssembly (50)) 23.5576923077
Projection-Wise Synapse Loss Projection ( PyAssembly (50) -> PyAssembly (50)) 23.182552504


Finally, the function plot_projectionwise_synapse_loss can be used to plot the lost and realized synapses of one projection.

def plot_projectionwise_synapse_loss(proj, marocco):
"""
plots the realized and lost synapses of a projection
params:
proj    - a pyhmf.Projection
marocco -  the PyMarocco object after the mapping has run.
"""
orig_weights = proj.getWeights(format='array')
mapped_weights = marocco.stats.getWeights(proj)
realized_syns = np.where(np.isfinite(mapped_weights))
lost_syns = np.logical_and(np.isfinite(orig_weights), np.isnan(mapped_weights))

conn_matrix = np.zeros(orig_weights.shape)
conn_matrix[realized_syns] =  1.
conn_matrix[lost_syns] = 0.5

import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
plt.figure()
plt.subplot(111)
plt.imshow(conn_matrix, cmap='hot', interpolation='nearest')
plt.xlabel("post neuron")
plt.ylabel("pre neuron")
plt.title("realized and lost synapses")
plt.savefig("synapse_loss.png")


## Details of the software stack¶

The BrainScaleS Wafer-Scale Software Stack is shown in Figure 112.

User-provided neuronal network topologies are evaluated by our PyNN API implementation (PyHMF), which is written in C++ with a Python wrapper. The data structures (spike trains, populations, projections, cell types, meta information, etc.) are implemented in C++ (euter). This layer also provides a serialization and deserialization interface for lower software layers. In a nutshell, euter serializes the PyNN/PyHMF-based experiment description into a binary data stream and hands over to the next software layer. In the following software layers, the translation from this biological neuronal network description into a hardware configuration will be performed. A large fraction of the translation work, in particular the network graph translation, is performed by the marocco mapping tool (described in the PhD thesis of S. Jeltsch. Code documentation is provided by doxygen and available here.

marocco uses calibration (calibtic) and blacklisting (redman) information to take into account circuit-specific properties and defects. This information is needed during the map & route process to homogenize the behavior of hardware neuron and synapse circuits and to exclude defective parts of the system.

The blacklisting works on the component level, e.g. denmems (the building blocks of neurons), on-chip buses, synapse drivers, etc. Denmems are blacklisted during calibration. If the calibration for any parameter could not be performed, the denmem is blacklisted. The number of available neurons, i.e. connected denmems, is not only depending on the number blacklisted denmems but also on their distribution. E.g. as an extreme example, if every second denmem would be blacklisted, neuron sizes larger than 1x2 (horizontal x vertical) are not possible.

The blacklisting for on-chip buses originates from the fact that every second horizontal and vertical bus is driven from a neighboring chip. If this neighbor chip can not be initialized, all buses from that chip must not be used.