number of strides per input pattern in algorithm.py

[piewchee]

Hi Sven,
@SvenGronauer
With reference to line 348-349 regarding the calculation of number of strides per input pattern.
windows = ((tensor_prev[0]-tensor[0])/stride + 1)**2*tensor_prev[2]
If I understand correctly , this parameter is used for calculation of total time required for simulation.
However I am confused with the above equation. wouldn't be the number of strides for one input layer
equal to tensor{0]*tensor[0] ?

[SvenGronauer]

Hello @piewchee ,
Yes, you are write: the quantity of windows can be used to calculate the total time required for the simulation.
I refer the size of one kernel shift as stride.
If you assume tensor[0]*tensor[0], then you would calculate the right number of windows for stride=1 (shifting the kernel over the input one step by one step).

But if you choose a stride > 1, then the number of windows is reduced with respect to the parameter stride.

Greetings!

[piewchee]

Hi Sven, @SvenGronauer Thanks for the reply. Isn’t it right that the number of windows should be reduced if the stride is >=2? By the way, I've tried your script with 100 training sample for each digit. I encountered problem with spinnaker with error msgs attached at the back of this reply. The spinnaker people suspects that the “input_neuron” is overloaded in a core. If I understand your script correctly, the input sample spikes are windowed according to kernel size (eg. 5x5). Each stride has simulation time of 100ms (and total_simtime=size_train_set*Sim_interval*windows). Isn’t that 100ms is long enough for kernel size of 5x5 spikes processing? Best Rgds PY 2018-09-26 06:33:47 INFO: *** Awaiting for a response from an external source to state its ready for the simulation to start *** 2018-09-26 06:33:47 INFO: *** Sending start / resume message to external sources to state the simulation has started or resumed. *** 2018-09-26 06:33:47 INFO: *** Awaiting for a response from an external source to state its ready for the simulation to start *** 2018-09-26 06:33:47 INFO: Application started - waiting 2186.134 seconds for it to stop 2018-09-26 07:10:19 ERROR: Error when calling spinn_front_end_common.interface.interface_functions.ApplicationRunner.__call__ with inputs {'loaded_application_data_token': True, 'no_sync_changes': 0, 'loaded_reverse_iptags_token': True, 'send_start_notification': True, 'txrx': transceiver object connected to 192.168.240.1 with 3 connections, 'send_stop_notification': True, 'loaded_routing_tables_token': True, 'loaded_binaries_token': True, 'notification_interface': <spinn_front_end_common.interface.interface_functions.notification_protocol.NotificationProtocol object at 0x7f27091db1d8>, 'buffer_manager': <spinn_front_end_common.interface.buffer_management.buffer_manager.BufferManager object at 0x7f270485a7a0>, 'executable_start_type': <ExecutableStartType.USES_SIMULATION_INTERFACE: (2, 'Calls simulation_run() and simulation_exit() / simulation_handle_pause_resume()')>, 'app_id': 18, 'time_threshold': 5, 'time_scale_factor': 1, 'wait_on_confirmation': True, 'executable_targets': <spinnman.model.executable_targets.ExecutableTargets object at 0x7f2712c54950>, 'loaded_iptags_token': True, 'runtime': 2186034.0} 2018-09-26 07:10:19 ERROR: An error has occurred during simulation 2018-09-26 07:10:19 ERROR: Operation waiting for cores to reach one of [<CPUState.FINISHED: 11>, <CPUState.PAUSED: 10>] timed out after 5 seconds Traceback (most recent call last): File "/home1/ncp/.local/lib/python2.7/site-packages/spinn_front_end_common/interface/abstract_spinnaker_base.py", line 1839, in _do_run executor.execute_mapping() File "/home1/ncp/.local/lib/python2.7/site-packages/pacman/executor/pacman_algorithm_executor.py", line 432, in execute_mapping self._execute_mapping() File "/home1/ncp/.local/lib/python2.7/site-packages/pacman/executor/pacman_algorithm_executor.py", line 448, in _execute_mapping results = algorithm.call(self._internal_type_mapping) File "/home1/ncp/.local/lib/python2.7/site-packages/pacman/executor/algorithm_classes/abstract_python_algorithm.py", line 44, in call results = self.call_python(method_inputs) File "/home1/ncp/.local/lib/python2.7/site-packages/pacman/executor/algorithm_classes/python_class_algorithm.py", line 56, in call_python return method(**inputs) File "/home1/ncp/.local/lib/python2.7/site-packages/spinn_front_end_common/interface/interface_functions/application_runner.py", line 111, in __call__ [CPUState.FINISHED, CPUState.PAUSED], timeout=timeout) File "/home1/ncp/.local/lib/python2.7/site-packages/spinnman/transceiver.py", line 2097, in wait_for_cores_to_be_in_state timeout) SpinnmanTimeoutException: Operation waiting for cores to reach one of [<CPUState.FINISHED: 11>, <CPUState.PAUSED: 10>] timed out after 5 seconds 2018-09-26 07:10:19 INFO: Attempting to extract data Getting Router Provenance |0% 50% 100%| ============================================================ Forcing error cores to generate provenance data |0% 50% 100%| 2018-09-26 07:10:19 ERROR: Unable to Finish getting provenance data. Abandoned after too many retries. Board may be left in an unstable state! ============================================================ Getting provenance data |0% 50% 100%| ============================================================ Extracting IOBUF from the machine |0% 50% 100%| ============================================================ 2018-09-26 07:10:19 ERROR: 0, 0, 2: RUNNING reverse_iptag_m 2018-09-26 07:10:19 WARNING: The reinjector on 0, 0 has detected that 15 packets were dumped from a core failing to take the packet. This often occurs when the executable has crashed or has not been given a multicast packet callback. It can also result from the core taking too long to process each packet. These packets were reinjected and so this number is likely a overestimate. 2018-09-26 07:10:19 WARNING: The callback queue for input_neurons:0:24 on 0, 0, 2 overloaded on 2008073 occasions. This is often a sign that the system is running too quickly for the number of neurons per core. Please increase the machine time step or time_scale_factor or decrease the number of neurons per core. 2018-09-26 07:10:19 WARNING: A Timer tick callback was still executing when the next timer tick callback was fired off for input_neurons:0:24 on 0, 0, 2, 2008088 times. This is a sign of the system being overloaded and therefore the results are likely incorrect. Please increase the machine time step or time_scale_factor or decrease the number of neurons per core 2018-09-26 07:10:19 WARNING: The timer for input_neurons:0:24 on 0, 0, 2 fell behind by up to 16 ticks. This is a sign of the system being overloaded and therefore the results are likely incorrect. Please increase the machine time step or time_scale_factor or decrease the number of neurons per core Traceback (most recent call last): File "main.py", line 96, in <module> training_function(rc, model) File "/home1/ncp/spinnaker/SpikingConvNet/training.py", line 112, in training_function s.run(network.total_simtime) File "/home1/ncp/.local/lib/python2.7/site-packages/spynnaker8/__init__.py", line 515, in run return __pynn_run(simtime, callbacks=callbacks) File "/home1/ncp/.local/lib/python2.7/site-packages/pyNN/common/control.py", line 110, in run return run_until(simulator.state.t + simtime, callbacks) File "/home1/ncp/.local/lib/python2.7/site-packages/pyNN/common/control.py", line 92, in run_until simulator.state.run_until(time_point) File "/home1/ncp/.local/lib/python2.7/site-packages/spynnaker8/spinnaker.py", line 128, in run_until self._run(tstop - self.t) File "/home1/ncp/.local/lib/python2.7/site-packages/spynnaker8/spinnaker.py", line 175, in _run AbstractSpiNNakerCommon.run(self, duration_ms) File "/home1/ncp/.local/lib/python2.7/site-packages/spynnaker/pyNN/abstract_spinnaker_common.py", line 312, in run AbstractSpinnakerBase._run(self, run_time) File "/home1/ncp/.local/lib/python2.7/site-packages/spinn_front_end_common/interface/abstract_spinnaker_base.py", line 927, in _run self._do_run(step, loading_done, run_until_complete) File "/home1/ncp/.local/lib/python2.7/site-packages/spinn_front_end_common/interface/abstract_spinnaker_base.py", line 1839, in _do_run executor.execute_mapping() File "/home1/ncp/.local/lib/python2.7/site-packages/pacman/executor/pacman_algorithm_executor.py", line 432, in execute_mapping self._execute_mapping() File "/home1/ncp/.local/lib/python2.7/site-packages/pacman/executor/pacman_algorithm_executor.py", line 448, in _execute_mapping results = algorithm.call(self._internal_type_mapping) File "/home1/ncp/.local/lib/python2.7/site-packages/pacman/executor/algorithm_classes/abstract_python_algorithm.py", line 44, in call results = self.call_python(method_inputs) File "/home1/ncp/.local/lib/python2.7/site-packages/pacman/executor/algorithm_classes/python_class_algorithm.py", line 56, in call_python return method(**inputs) File "/home1/ncp/.local/lib/python2.7/site-packages/spinn_front_end_common/interface/interface_functions/application_runner.py", line 111, in __call__ [CPUState.FINISHED, CPUState.PAUSED], timeout=timeout) File "/home1/ncp/.local/lib/python2.7/site-packages/spinnman/transceiver.py", line 2097, in wait_for_cores_to_be_in_state timeout) spinnman.exceptions.SpinnmanTimeoutException: Operation waiting for cores to reach one of [<CPUState.FINISHED: 11>, <CPUState.PAUSED: 10>] timed out after 5 seconds From: Sven Gronauer [mailto:notifications@github.com] Sent: Saturday, 29 September, 2018 5:59 PM To: SvenGronauer/SpikingConvNet Cc: Chee Piew Yoong; Mention Subject: Re: [SvenGronauer/SpikingConvNet] number of strides per input pattern in algorithm.py (#4) Hello @piewchee<https://github.com/piewchee> , Yes, you are write: the quantity of windows can be used to calculate the total time required for the simulation. I refer the size of one kernel shift as stride. If you assume tensor[0]*tensor[0], then you would calculate the right number of windows for stride=1 (shifting the kernel over the input one step by one step). But if you choose a stride > 1, then the number of windows is reduced with respect to the parameter stride. Greetings! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#4 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Ad0d2PtgY_yhQeH5w3Y1Ayp78l-45nQMks5uf0R5gaJpZM4WvQwo>. This e-mail and any attachments are only for the use of the intended recipient and may contain material that is confidential, privileged and/or protected by the Official Secrets Act. If you are not the intended recipient, please delete it or notify the sender immediately. Please do not copy or use it for any purpose or disclose the contents to any other person.

[SvenGronauer]

Hello @piewchee ,

1.) Yes, that's true. If you increase the stride >=2, the number of windows is reduced.

2.) Have you changed any machine parameters (like machine time step or time_scale_factor or do you have too many neurons per core ?) I used 100 neurons per core, and the other two parameters set to default.

Otherwise, try to change these parameters in parameters.py to the following values:
SIM_INTERVAL = 50 IDLE_FIRING_TIME = 25

3.) In my script, each kernel is given a time window of 100ms (=sim_interval) to get potentiated by input neurons. After the 100ms sim interval, the kernel is shifted and the next neurons are potentiated.

[piewchee]

HI @SvenGronauer
Yup I tried the reduced the timing. I did not change the 100 core limit.
The spinnnaker ppl think that the input_neurons in the same core fire at the same time step may have cause the overload. I noticed that there is a possibility of 5 neurons (out of 9 in 3x4 kernel) fire at the same time-step.
By the way, i noticed there is something strange with the rate_code spiketimes.
I understand that if intensity > threshold, it will fire 10 times randomly with t_d and offset.
However,
for eg. neuron id_0[ the spiketimes is the same for 10 times. eg. 54900, 54900, 54900 ..]

Rgds
Del

[SvenGronauer]

Hi @piewchee,
the amount of spikes is not exactly 10 times.
If the the intensity of the input pixel as greater than a threshold, then the amount of spikes should be proportional to the input pixel intensity. Do you present the input pixels (input image) as uint8 in the interval [0,255] ?