Minicolumns: The W.Schultz Task

Previous: Minicolumns: The Thorpe Task

Contents:

• Task Protocol
• Modifications of the Task
• Model Construction
  • Simulating the environment of the Schultz Task
  • Integrating environment output during the task with the State-Action pair generating circuitry
  • Using prefrontal minicolumn output to guide go-actions during task performance
• Results of task performance
• Comparison with experimental results published by W.Schultz et al.
• Future Directions

Task Protocol

Figure 1: The protocol of the W.Schultz task. Solid arrows indicate the possible steps taken within a trial. Dashed arrows indicate possible transitions between consecutive trials.

Minicolumns required for this task are the following:

• STATE-S-rewarded-move
• STATE-S-rewarded-non-move
• STATE-S-unrewarded-move
• STATE-reward
• ACTION-go
• ACTION-no-go

Modifications of the Task

• The trigger state is not included: The same trigger was used after all stimuli to elicit move or non-move responses. The same behavior may be achieved in the simulation without a separate reponse trigger.
• The sound state is omitted: A sound was used for conditioning in unrewarded-move trials. Since tbose trials must lead to learned association with subsequent rewarded trials, the explicit sound state is left out here to simplify computations in the simulation and to maintain a strict protocol of STATE-ACTION sequences even in the long association paths formed following an unrewarded-move trial.
• As in the simulation of the Thorpe task, time intervals in the stimulus protocol are scaled down.

During the training presentations, randomized go/no-go actions are simulated. During testing, go/no-go actions are controlled by the neuronal simulation.

Model Construction

Simulating the environment of the W.Schultz Task

The task reversal mechanism that was included for the Thorpe task is removed. These modifications are implemented in minicolumns-schultz-task.20040114.ccm. Since, reward is now the fourth state, the connection table "goal" is updated accordingly, so that retrieval can activate the reward minicolumn when a desire for reward is generated. A second route is added to the reward generating circuitry, so that non-move is rewarded when the rewarded-non-move stimulus is presented.

Figure 2: Reward generating circuitry in the W.Schultz task. Reward may be given six rhythmic cycles after the correct action follows presentation of a rewarded-move or rewarded-non-move stimulus. Trains of reward spikes elicited by action spike trains are gated by the presence of the corresponding stimulus. Additionally, stimuli can only control transmission to the gating neurons 750 ms after a move (go) or non-move (nogo) action is commenced. Stimulus presentations are again relieved of Control of the gating neurons once buffer-clear signals appears between trials.

The following table lists a proposed pseudo-random set of training trials that enable all relevant associations to be learned.

trial start cycle	cycle offset	cycles new a	stimulus	cycle offset	cycles new a	action	cycle offset	cycles new a	consequence	cycle offset	inter-trial	new associations
1	0 (125ms)	0-2	unrewarded-move	0 (125ms)	3-5	nogo				6 (875ms)	clear	unrewarded-move to nogo
8	0 (1000ms)	0-2	unrewarded-move	0 (1000ms)	3-5	go						unrewarded-move to go
14	0 (1750ms)	0-2	rewarded-move	0 (1750ms)	3-5	go	6 (2500ms)	6-10	reward	11 (3125ms)	clear	go (from unrewarded-move) to rewarded-move to go to reward
26	0 (3250ms)	0-2	rewarded-move	0 (3250ms)	3-5	nogo				6 (4000ms)	clear	rewarded-move to nogo
33	0 (4125ms)	0-2	unrewarded-move	0 (4125ms)	3-5	go
39	0 (4875ms)	0-2	rewarded-non-move	0 (4875ms)	3-5	nogo	6 (5625ms)	6-10	reward	11 (6250ms)	clear	go (from unrewarded-move) to rewarded-non-move to nogo to reward
51	0 (6375ms)	0-2	rewarded-non-move	0 (6375ms)	3-5	go				6 (7125ms)	clear	rewarded-non-move to go

Table 1: A minimal set of training trials to encode all relevant associations. Each cycle has a period of 125 ms, so that the minimum amount of simulated time needed during training is 7250 ms (58 cycles). If more than one presentation of each association is needed that time increases accordingly. Absolute times indicated in milliseconds in the second column are entered into the stimulus event sequencer together with the stimulus index. Absolute times in the fifth column are entered into the action training event sequencer together with the action index. Absolute times in the eigth column are automatically computed by the reward generating circuitry. And absolute times in the eleventh column are entered into the clear-buffer event sequencer with several repetitions at 10 ms intervals. This training protocol is implemented in minicolumns-schultz-task.20040115b.ccm.

Integrating environment output during the task with the State-Action pair generating circuitry

The Catacomb stimulus objects activate and deactivate in response to the same spike index, a toggle switch. Thus, the onset of a stimulus presentation must set up switches such that a clear-buffer signal is interpreted as the corresponding spike index. Also, the toggle switch must receive only the first clear-buffer spike. That second condition can be met either by propagation through a neuronal population with after-hyperpolarization or through further switching circuitry. Switching circuitry is as desirable as a neuronal solution, since this construct belongs to the simulation of the environment task protocol.

There is currently a Catacomb related problem with the control of transmission modulation that affects task environment circuitry intended to signal the end of stimulus presentations during trials (minicolumns-schultz-task.20040115c.ccm). In minicolumns-schultz-task.20040115d.ccm the problem is temporarily avoided by manually coding those signals.

Figure 3 shows the spike trains during the training phase of the task, while figure 4 shows the corresponding spike pairs that are sent to the minicolumns of the prefrontal cortex model.

Figure 3: Spike trains produced during training in the W.Schultz task simulation. Spikes indices represent: (0) go, (1) nogo, (2) rewarded-move stimulus, (3) rewarded non-move stimulus, (4) unrewarded move stimulus (5) reward. Notice that reward appears immediately in the third trial. The simulation circuitry includes a 750 ms delay of reward compared to the onset of action, but the action was not interrupted by a clear-buffer signal between the second and third trials. A similar issue appears between the fifth and sixth trials.

Figure 4: Action-state spike pairs generated in specialized neuronal circuitry for spike trains representing states, perceived stimuli and reward received, and representing actions, go/no-go. There are missing stimulus state spikes and no reward state spikes. Some of these problems may be caused by the issues noted in Figure 3.

In order to correct the onset of the reward spike train, the input connection that provides 750 ms delayed switching spikes is no longer connected to action events, but rather to the output of the event sequencer that specifies the start of each stimulus presentation. The router connection table is updated accordingly.

Figure 5: Spike trains produced during training in the W.Schultz task simulation. Spikes indices represent: (0) go, (1) nogo, (2) rewarded-move stimulus, (3) rewarded non-move stimulus, (4) unrewarded move stimulus (5) reward. Spike trains representing reward received is now correctly delayed 750 ms after the onset of a training trial.

Figure 6: Action-state spike pairs generated in specialized neuronal circuitry for spike trains representing states, perceived stimuli and reward received, and representing actions, go/no-go. The spike indices represent: (0) go ACTION, (1) nogo ACTION, (4) rewarded move stimulus STATE, (5) rewarded non-move stimulus STATE, (6) unrewarded move stimulus STATE and (7) reward STATE.

Inspection of the a population activity in the minicolumns shows whether all associations listed in the last column of Table 1 appear together within an interval that can elicit LTP.

Figure 7: Activity of the minicolumns a population during training in the W.Schultz task. Membrane potentials plotted at the top represent states, while those plotted at the bottom represent actions. States are color coded as follows: (BLUE) unrewarded move stimulus state, (RED) rewarded move stimulus state, (CYAN) reward state and (GREEN) rewarded non-move state. Actions are color coded as follows: (MAGENTA) nogo action and (BROWN) go action. At some time, each of the associations listed in the last column of Table 1 appears in the a-STM-buffer in correct order.

Using prefrontal minicolumn output to guide go-actions during task performance

		to r population
		go	no-go	Srm	Srnm	Surm	reward
from s population	go			(3)	(6)		(3)
	no-go						(6)
	Srm	(3)	(4)
	Srnm	(7)	(6)
	Surm	(2)	(1)
	reward

Table 2: Connections strengthened in Wf between the s and r populations of minicolumns. Strengthened connections are indicated by bracketed numbers that represent the first training set in which the association is learned (see Fig.6). The associations that are needed to achieve the task are highlighted in blue with bold font. The strengthened connections listed by numerical indices are Wf:{4,5,6,12,13,14,32,40,56,57}. While LTP appears to be elicited at the correct synapses, it is possible that the strength achieved after a single batch of training presentations is insufficient for successful retrieval. It is also possible that propagation needed for encoding in Wif is not achieved with a single batch of training presentations.

		to x population
		go	no-go	Srm	Srnm	Surm	reward
from y population	go			(3)	(7)	(2)
	no-go			(4)	(6)	(1)
	Srm	(3)
	Srnm	(6)
	Surm
	reward	(3)	(6)

Table 3: Connections strengthened in Wb between the y and x populations of minicolumns. The notation is identical to that in the previous table. The strengthened connections listed by numerical indices are Wb:{4,5,7,15,32,33,40,41,48,49}.

By driving the relevant neurons in the x population with an event sequencer in multiple simulation runs, two tests at a time (at t=7300 ms and t=7350 ms), the following results were obtained for encoding of Wib after a single batch of training presentations:

• go minicolumn:
  • x{4} to y{6}: Srm to Surm strengthened
  • x{5} to y{6}: Srnm to Surm strengthened
  • x{7} to y{6}: rew to Surm incorrectly strengthened
• nogo minicolumn:
  • x{15} to ? not encoded
• Srm minicolumn:
  • x{32} to ? not encoded
  • x{33} to ? not encoded
• Srnm minicolumn:
  • x{40} to ? not encoded
  • x{41} to ? not encoded

The single batch of training presentations was probably insufficient to create strong LTP and to establish LTP in Wib and Wif. One way to improve this is to increase the learning rate. A more plausible solution is to provide more Hebbian spiking at the relevant connections. That can be accomplished by running another batch of stimulus presentations or by allowing each pair of spikes to persist longer in the initial batch of stimulus presentations. Longer persistence requires only a few additional cycles per item, while another batch would require many more cycles. For reasons of computational efficiency I therefore opt to increase the intervals between successive items. For this, the data protocols in event sequencers as well as several delay buffer parameters are modified.

The first modification applied in minicolumns-schultz-task.20040116b.ccm increases the interval between state and action spikes in each state-action spike pair, thereby providing additional time for encoding. Modifications:

• The action spike delay is increased by 250 ms to 625 ms.
• The reward delay is increased by 250 ms to 1000 ms.
• Stimulus presentation events are shifted to add 250 ms between eaech trial.
• Presentation end events are shifted to add 250 ms between each trial.
• Action events are shifted to add 250 ms between each trial.
• Buffer-clear events are shifted to add 250 ms between each trial.
• Runtime is increased by 1750 ms to 9050 ms to span seven training trials of increased duration.

Figure 8: Spike trains produced during training in the W.Schultz task simulation. Spikes indices represent: (0) go, (1) nogo, (2) rewarded-move stimulus, (3) rewarded non-move stimulus, (4) unrewarded move stimulus (5) reward.

Figure 9: Action-state spike pairs generated in specialized neuronal circuitry for spike trains representing states, perceived stimuli and reward received, and representing actions, go/no-go. The spike indices represent: (0) go ACTION, (1) nogo ACTION, (4) rewarded move stimulus STATE, (5) rewarded non-move stimulus STATE, (6) unrewarded move stimulus STATE and (7) reward STATE.

The resulting strengthened connections in Wb and Wf are the same as in tables 2 and 3 above, with one additional strengthened connection in each, representing an unexpected association from nogo to Srnm. This is probably caused by an error in the clear-buffer signals around t=9750 ms that do not correctly clear the action maintaining circuitry. Adding some more clear-buffer signals after t=9750 ms removed the undesired association between trials in minicolumns-schultz-task.20040121.ccm. A nogo action spike still appears sfter reward just before the end of the sixth trial. That spike could be removed by starting the clear-buffer signals a little earlier, but it is allowed to occur, since that is not in conflict with the method of encoding state-action pairs in the W.Schultz task.

Results of task performance

Figure 10: Reward is successfully obtained during performance of the W.Schultz et al.. go/nogo task with the encoded minicolumns model of prefrontal cortex in minicolumns-schultz-task.20040121b.ccm. As noted in the plot of spike trains, spike indices represent the following: (0) go action performed in response to minicolumn output, (1) nogo action performed in response to minicolumn output, (2) rewarded-move stimulus perceived, (3) rewarded non-move stimulus perceived, (4) unrewarded move stimulus perceived, (4) reward obtained. Training ends at t=11375 ms. Thereafter, a desire for reward exists and each of three possible trial conditions is given (the third appears twice, since its successful performance can randomly lead to two different rewarded trials). At t=11750 ms, the rewarded move stimulus appears. Shortly thereafter, retrieval in the network of minicolumns correctly causes a go action response. That response is then rewarded. Similarly, rewarded non-move stimulus at t=13750 ms elicits the correct nogo action. The unrewarded move stimulus appears for the first time at t=15750 ms. Since retrieval causes the desired go action, a rewarded move trial follows at t=17750 ms. Finally, the other possible rewarded conclusion of a successful unrewarded move trial is demonstrated at t=19750 ms, namely a following rewarded non-move trial at t=21750 ms.

Figure 11: Spikes in the a population of minicolumns during performance of the W.Schultz et al.. go/nogo task. Spike color codes indicate the following minicolumns: (RED) rewarded move, (GREEN) rewarded non-move, (BLUE) unrewarded move, (CYAN) reward, (BROWN) go and (MAGENTA) nogo. The continuous presence of reward spikes (CYAN) during task performance represents the ever present desire for reward in terms of reward minicolumn activation during retrieval phases. When the perception of reward obtained activates the reward minicolumn during encoding phases at the end of successful trials denser clusters of reward spikes appear. When compared with the onset of go/nogo action spikes in figure 10, activity in the a population of go/nogo minicolumns is delayed by a few cycles. This is caused by time taken during operations in the loop from the minicolumn output population activity that causes go/nogo action to spiking at the new item phase of the a-STM-buffer that is propagated to the a population.

Note:
In minicolumns-schultz-task.20040122.ccm, results displayed unusual gaps in the action spike train (and therefore the dependent reward delivery spike train) after t=13410 ms after t=15415 ms. The gaps are shown in the figure below. Inspection reveals that the minicolumn output population that drives action spike trains during task performance experiences a single spike at cells 56 to 63 around those times. Those cells belong to the reward minicolumn. Further investigation is needed to discover why this happens, and why it happens only so sporadically.

Figure 12: Undesired gaps show up in the action spike trains after t=13410 ms and t=15415 ms. Since the spike train of reward delivery in the environment simulation depends on the action spike train, the gaps show up there as well.

Comparison with experimental results published by W.Schultz et al.

(follow link for full size)

Model correspondence with results in fig.13a:
Activity in (A) may correspond to the activity of neurons in Srm, Srnm and Surm minicolumns (in a, x, y, r or s populations).

• good: We show activity at the stimulus time in specific minicolumns.
• bad: W.Schultz et al. show a reduction of activity (almost none) during the 2 second interval between stimulus presentation and the decision to go/nogo (trigger). A similar result could be accomplished with our model if the "delay" is specifically encoded with a separate minicolumn. But, W.Schultz et al. did not report neurons firing selectively during the delay! Thus, the difference may be an issue particular to the neuron recoded by W.Schultz et al., or it may point to an issue with the proposed mechanisms in the model.

Model correspondence with results in fig.13b:
The neuron may correspond to a neuron in the go/nogo minicolumns of our model (in a, x, y, r or s populations).

• good: We have such specific activity.
• good: W.Schultz et al. also show continuous firing once go/nogo is activated ("predictive" / preceding reward).
• good: Our go/nogo also continues during reward.

Model correspondence with results in fig.13c:
Reward neurons (in a, x, y, r or s populations)

• good: specific.
• good: continuous.
• good: dense; desired (retrieval) and obtained (encoding).

There is a problem with the current simulation output! The behavior is correct in that go/nogo is controlled properly, but the go neurons are not activated with enough specificity.

PROBLEM:

During retrieval phases, r is being activated by y regardless of Srm or Surm trial type, which should not happen. This could be due to either too much strength in y->r connections or Wf connections during retrieval. The Wf option is not as likely, since activation due to forward propagation without sufficient gating probably would not lead to spikes in r from Surm when Srm is the current state. The r neurons that are active also match perfectly the y-diffuse neurons that are active. Note that Srm(Wf)->r(Go)=r(4), Surm(Wf)->r(Go)=r(6). And r receives input from y-diffuse and via Wf. It sends output to the r STM buffer, not driven by it. The y connection appears to be 2.5nS, which should be insufficient on its own. So there must be a contribution via Wf to r(Go) from both Srm and Surm to cause r(4) and r(6) to spike. That means that s(Srm)->Go=S(32) and s(Surm)->Go=s(48) must both be spiking. It looks like s(Srm)->Go, s(Surm)->Go and even s(Srnm)->Nogo are spiking during the Srm trial.

Q1: Is s being driven directly by x? Transmission strength from x to s is halved at the same phase as transmission strength to r via Wf. But that is half (or a little more than half when modulation is not at its lowest) of 6nS. The very strong connection exists, because x has to drive s during encoding. During Srm trial, false r(6) spiking needs support from y(6) and from s(48). y(6) depends on x(4) (from Srm) or x(5) (from Srnm). Even with end-stopping, x(5) can be activated by y(40) (in Srnm), since that is on another propagation path from the goal minicolumn. So, the y(6) contribution is both valid and present. s(48) [IS IT REALLY THERE? OR IS Y DRIVING?] in Surm cannot be driven by current state input during a Srm trial. s(48) should depend on both x(48) and on an r in the r(48-55) range, but no such Wif connection should exist, since nothing ever precedes Surm. The output shows no r(48-55) activity during the Srm trial, hence s(48) must be driven directly by x(48)! There is only one issue here: x should not be able to drive s during retrieval, sincex(48) depends on y(6) and y(6) activity was already explained above.

Q2: If Q1 is true, that explains why s(Srnm)->Nogo is spiking in response to propagation from the reward goal, but why does current Srm activity not cause end-stopping that prevents s(Surm)->Go spikes? Current state input is properly connected to the interneuron population that is responsible for end-stopping. And it appears to be modulated at the same phase as the y-specific to y-diffuse connection, i.e. to function during retrieval. And the end-stopping synapse is quite strong. End-stopping appears to be in place. End-stopping is probably not the issue, as explained for the case of y(6) activity in Q1.

Transmission modulation halves the input strength from Wf to r during retrieval (perhaps test this!). If not, s alone might drive r. SOLUTION: (1) extract the spike trains according to the responses they SHOULD exhibit in our model (2) mark the bug in the graphs that need to be changed, check all graphs (3) focus on completing the Results, Discussion and Abstract text (4) while correcting the text for submission according to comments, and while doing other tasks simultaneously, correct the model and its output and update the figures correspondingly (Due to the answer to Q1, perhaps the modulation of transmission from x to s needs to be down to 0.3 instead of 0.5. This solution is attempted, but not yet tested in minicolumns-shultz-task.20040401.ccm)

After changing the transmission modulation, behavior is still correct, but r(6) still spikes in the retrieval phases of an Srm trial. Even after reducing the minimum of the transmission modulation to 0, the problem remains. Either the transmission modulation is not being applied at the right phase, or something else is driving r(6) too strongly.

r(6) input ONLY from y(6) and s(48) s(48) input from x(48) and r(48-55) and current state input, but that is zero during a Srm trial r(48-55) input is a possible cause, but should be weak even then, check if it is zero during a Srm trial: those neurons never spike (which is correct) I can test if y(6) or x(48) are driving gated neurons directly by explicitly cutting other input (without explicit cutting:) both are active at the times when r(6) is driven during retrieval in a Srm trial (with explicit cutting:) if transmission modulation is not enough to control the amount of direct drive then I can also use modulation of the membrane potential

after cutting input other than x to s, x(48) and s(48) still spike during retrieval phases of a Srm trial and r(6) is still spiking in what seems the same way

also cut other input to r!!! then analyze what this means

when forward input to r is cut there is no more spike activity in r, so r is never driven entirely by y - thus y has the appropriate connection strength to r and the false activity in r(6) is dependent on s(48) but s(48) is still active, which is therefore driven directly by x(48) ~/src/nnmodels/ccmb/minicolumns-schultz-task.20040408.ccm.gz was used for this testing, although corrections should revert to minicolumns-schultz-task.20040401.ccm

Inspection of a and of the input that triggers retrieval shows that the retrieval spikes are the ongoing spikes in x(48), which disappear only once after t=12600 ms, when the newly appearing encoding spike is too proximate. The retrieval spikes fall just within the high portion of transmission modulation. The transmission modulation drops slightly too late. A reduction of the delay of transmission modulation by about 6-7 ms may solve the problem. Subsequently, the minimum level of the modulated transmission strength must be set to enable correct retrieval.

In minicolumns-schultz-task.20040420b.ccm, the delay of transmission modulation is reduced by 10 ms (actually, the delay is removed entirely). This modification affects transmission modulation of input via Wf to r and via Wif to s. This was successful, and r(4) and r(6) no longer produce the same spike trains. r(6) is now active only in encoding phases during the Srm trial.

Increasing the similarity of model output with that of the Schultz et al. results:
In order to further increase the visual similarity of the output, I will for the time being curtail forward propagation beyond the single step. That means that the forward propagation cannot be used for planning in this simulation, a feature that is not needed for the Schultz et al. task.

The approach adopted in minicolumns-schultz-task.20040711.ccm is to link to the r and s populations separate populations of interneurons (represented by a single interneuron component). Those experience population activity in response to one ore more firing r or s population neurons. The connectivity from the interneuron populations to the r and s populations is such that activity in one minicolumn sends inhibition to all other minicolumns. (In principle, the inhibition can simply target all minicolumns, so perhaps I can simply use such connectivity.) The inhibition is desired only during retrieval modes, or during the performance stage of the task. To do this, I do the following steps:

1. Add an interneuronal population for r and one for s. Note that what we want now has less to do with winner-take-all responses than responses only in the minicolumn associated with current state.

   Both r and s populations already have links to interneuron populations. So, I will reuse those at present. I will convert the interneuron populations from representations with 8 cells to representations with a single cell, and I will replace the UseinvWa input routing to the interneurons and UseWa output routing from the interneurons with simple all-all connections. In order not to receive inhibition at this time from the STM buffers that are intended for future implementations of p and q populations, I will also cut the UseWa connections to r and s from the interneuron populations driven by those buffers. I have set the conductivity in the GABAergic synapses of r and s to 5 nS.

2. Add all-all connectivity (test this and see if specific routing is needed).

3. Add after-hyperpolarization.

4. Add GABA_B modulation of synaptic transmission to the interneuron populations. Initially drive that modulation so that transmission is only successful during retrieval modes. If that does not achieve the desired effect, drive it instead so that successful transmission occurs only during the performance stage of the task. (I'm actually trying the second option first, simply as a system test, since it is assured to interefere less with learning.)

5. Check if the task is still properly learned and if performance is successful.

   Using the transmission modulation driven according to the stage of the task, performance is moderately successful. Unfortunately, performance degrades as more trials are done. This is caused by the effect of the inhibition on the encoding phases that go on during the performance stage of the task. Thus, it is shown that there are the following ways to achieve the desired responses:

   (1) Suppress transmission of inhibition well during the encoding mode of each theta cycle.
   (2) Separately control STDP and transmission during the learning and performance stages of the task. Both STDP and transmission must then be enabled during learning, while the performance stage involves lateral inhibition in populations r and s, as well as suppression of plasticity.
   (3) Instead of suppressing the firing of other neurons in r and s populations, suppress the transmission along the paths from current state during the retrieval mode of each theta cycle.
   (4) Instead of suppressing the firing of other neurons in r and s populations, suppress the transmission along the paths from current state during the performance stage of the task. In this case, firing can still occur as needed during the encoding modes of each theta cycle in the performance stage of the task.

   The drawbacks of approaches (3) and (4) are the reliance on transmission modulation. While other approaches also involve such modulation in the current implementation, the effect of that modulation can often also be achieved by adding modulation of membrane potential to target neurons, which affects their tendency to fire. That may also be possible for (3) and (4), but may involve further changes in the strength of transmission used during encoding modes so that the ability to fire during encoding is unaffected. The advantage of approaches (3) and (4) is clearly that the modulation directly targets the functional need: A desire to restrict propagation. Other features, such as neuronal spiking in other minicolumns that is caused by other input are left unchanged.
   Approach (2) has the drawback that it relies on the modulation of plasticity, a phenomenon that we used in models of hippocampal function, but one that we have not yet addressed in terms of physiological evidence in PFC.
   I would prefer to try (1) now.

   But there is one more issue: Since there is no LTD in the current model, and new connections should not appear in Wf when r and s neurons in only the current state minicolumn can fire during task performance, how does the degradation of performance actually happen? Answer: The output population produces spikes when a combination of current state input and output from the x population is received. This does not depend upon Wf (which is correct). Therefore the difference must lie in the x population output. That was confirmed by inspection of x spiking with and without the inhibition of r and s population activity. The only time when r or s population activity is relevant to the x population is during encoding. This means that proper encoding of associations, specifically those associations involved in Surm trials (Go action leads to Srm or Srnm trials), was not completed at the end of the learning stage of the task. Apparently, previous simulation runs completed that learning during the performance stage so that the issue did not appear in the results. The possible solutions: (a) Increase the duration of the learning stage of the simulation. (b) Allow activity to propagate unhindered during the encoding mode of each rhythmic cycle. I choose (b), since simulations already take quite long and since changing the task protocol would involve recalculating a large number of spike event times for the event sequencers. This choice works well with alternative (1) above.

   These changes are all implemented in minicolumns-schultz-task.20040712.ccm.

6. Check to see if the neuron activity is now more easily recognizable as that seen in the Schultz et al. results.

7. Use the standard method to extract figure data and produce the comparison figure.

8. Consider adding noise.
   I have made an attempt to add noise to the entire system by replacing the perfectly regular 8Hz rhythm provided by a signal generator with a circuit that produces a rhythm that has some added noise. This is implemented in minicolumns-schultz-task.20040715.ccm.

To create improved figures of a-STM-buffer repetition and replacement, I made the model implementation minicolumns-schultz-task.20040715b.ccm, which produces responses in the vector recorder that are processed with the Yorick script showstacked.i.

Now, I am attempting to send the desire for reward directly to the y population instead of through the a population so that a(rew) data will look cleaner. This is implemented in minicolumns-schultz-task.20040717.ccm. Some further care needs to be taken with regard to the output to x and y populations from a, since the modification initially results in a loss of performance.

As it turned out, I had made the wrong modification by removing output from the a population to the y population. Instead, I had to remove the "desire for reward" input to the a population and send that directly to the y_diffuse population with the UseWa connection router table. That led to correct performance and an improved match between the responses of our a population and observations made by Schultz et al. As a result of the change in the model, the output for task performance has also become somewhat clearner. New data is stored in minicolumns-schultz-task.20040717.a-population.cdf.gz and minicolumns-schultz-task.20040717.task-performance-results.png. That was used to update figures as described in TL#200407190918.

Despite the fact that inhibition is used to suppress propagation through through r and s populations and to achieve one-step activity in s, there appears to be regular activity in some r and s neurons. The following analysis attempts to determine the cause.

              retr.  retr.  enc.   enc.   retr.  enc.
s7(Go->Rew) : 13146, 13264, 13364, 13478, 13526, 13600
r56(Rew<-Go): 13148, 13266, 13366, 13480, 13528, 13602
a5(Go)      : 13128, 13228, 13335, 13451, -----, 13573
a4(Rew)     : -----, 13255, 13356, 13469, -----, 13590
state(Go)   : there is none, since Go is an action 
r4(Go<-Srm) : 13142, 13243, 13352, 13466, 13540, -----
r5(Go<-Srnm): 13142, 13243, 13349, 13466, -----, -----
r6(Go<-Surm): 13142, 13243, 13352, 13466, -----, -----
x7(Go<-Rew) : 13143, 13261, 13360, 13475, 13522, 13596
                            FINE   FINE          FINE

• r is clearly dependent on s here, i.e. it is probably not driven directly by y or anything like that, but it is propagated. But that could be one-step if s is activated by something local to that minicolumn e.g. if s is driven by x then gating by y also allows r to spike.

• Both a(Go) and a(Rew) spikes precede s and r spikes, so they may be driving influences, but the delay is quite considerable and they do not precede all instances of r and s activity, such as the 13526/8 instance. There is no direct connection from a to r or s, but s does receive state input from outside.

• During retrieval, r and s are gated by y and x.

• s(Go->Rew) does not appear to be driven by the r(Go) neurons, since there are several instances where r(Go) spikes are absent or later than those in s(Go), although in the cases where r(Go) spikes do precede s(Go) spikes by some time difference, they may be involved in combination with the timing of the gating x(Go) input.

• Is x driving s? It certainly looks as if s(Go->Rew) is being driven directly by x(Go<-Rew). Yet, s should only be DRIVEN by x during encoding.

• Which mode are the r and s spikes in, encoding or retrieval? Three of the spikes are in retrieval modes and three in encoding modes it makes sense that x drives s during the encoding modes. But note that all the s(Go->Rew) spikes between t=16300 and t=17600, a whole regular span, fall precisely within retrieval modes and the only explanation for their appearance is input from x(Go<-Rew).

• As it turns out, the x population retrieval spikes are transmitted at full strength, since the transmission modulation of the x to s connection does not have the necessary phase offset. A proper phase offset may be 95 ms. This solves the previous mystery of x driving s as well (see above), and allows the model to be corrected. Since transmission was modulated at the wrong phase offset, it is surprising that x driving s worked well enough to achieve learning. There may therefore be some differences in the learning or performance output after applying the correction to the simulation. The corrected was applied in minicolumns-schultz-task.20040720.ccm

  Unfortunately, the correction (while desirable) causes incomplete performance. The previous "bug" in the r and s forward circuitry enabled some learning to occur during the performance stage of the experiment. With the corrected model, that learning must be accomplished before correct performance is possible, which would require a change in the experimental protocol and longer simulation times. For this reason, and because the forward retrieval is not essential to function in the present replication of Schultz et al. results, the corrections will not be used for the schultztask paper. The results published there will consequently focus little on the r and s populations. Time invested in this analysis and the model correction is not wasted if the minicolumn PFC model is used again in future simulations..

  It appears that the previous attempt at a solution may have caused a different set of oddities, simply by comparing spike times of x and s with the wrong transmission modulating response. In any case, the delay that had previously been used before the first attempt at a solution was intended to produce 0.5 transmission through Wf during retrieval and 1.0 transmission through Wf during encoding. That delay therefore closely matched the delay of 9 ms of current state input, even though this may have been wrong (tests follow). Thus, in both cases x to s and r via Wf, transmission should be reduced during retrieval and increased during encoding.

In minicolumns-schultz-task.20040721.ccm.gz, I attempt to improve the flexibility of the minicolumn PFC model by taking the minicolumns out of their capture box. Further clean-up is done in minicolumns-schultz-task.20040726.ccm.gz, in which the vector recorder output of population a is returned to its normal order, equal to that of the minicolumns:

0	0-7	Go
1	8-15	NoGo
2	16-23	-
3	24-31	-
4	32-39	Srm
5	40-47	Srnm
6	48-55	Surm
7	56-63	Rew

(The first number indicates the minicolumn ID number, the range of numbers after that indicates the corresponding group of neurons in populations of 64 neurons.) In minicolumns-schultz-task.20040727.ccm.gz, some unused stimulus control circuitry is removed to clarify the "Stimulus-Presenter" and "Presentation-End" event sequencer control of stimuli in the task. This follows a detailed description of event sequencer function in the experiment protocol that will be used to create a computer aided method of rapidly modifying the protocol in a sensible manner.

A method that aids in the modification of the experiment protocol is now implemented in the set of scripts protocol.sh (this script calls the other two), protocol.m and the C++ utility protocol.cc. As an example, a few minor modifications of the protocol were made in minicolumns-schultz-task.20040728.ccm.gz. And in the example minicolumns-schultz-task.20040728b.ccm.gz, the duration of the learning stage is doubled by duplicating the set of learning trials and moving the performance stage accordingly (protocoldata20040728.m). Note that the use of the protocol utility bumped into a limitation of Catacomb with regards to the number of events in an event sequences.

Future Directions