Learning of hidden state t=0

[peroyose]

Dear all,

to my knowledge it's at the moment not possible to train the initial state of a recurrent layer (rec_layer.py).
For same applications this is helpfull and quite common. Therefore this pull request.

Thanks in advance for your answer,
Christian

P.S.:
I try to implement this very roughly in the "class RecurrentLayer". But it seems not so easy as expected. I introduced a new parameter (added to self.params), but the problem is training with mini-batches. Broadcasting seems not to work but mayby I made other mistakes. The function does't compile

initialization

    if self.train_init_state:
        # trainable init stated at t=0                                                                                                   
        self.init_state = theano.shared(
            numpy.zeros(self.n_hids, dtype=theano.config.floatX),
            name="init_state_%s"%self.name)
        self.params.append(self.init_state)

in method fprop:

    if not init_state:
        if hasattr(self, 'init_state'):
            if not isinstance(batch_size, int) or batch_size != 1:
                # not possible with shared or symbolic variables in theano                  
                #init_state = TT.tile(self.init_state, (batch_size, 1))
                # don't work and will not store the new                                                               
                #init_state = TT.alloc(self.init_state, batch_size, self.n_hids) 
                # broadcasting does't work ??                                    
                init_state = self.init_state
            else:
                #TODO test                                                                                                               
                init_state = self.init_state
        else:
            assert self.train_init_state == False
    if not init_state:
        if not isinstance(batch_size, int) or batch_size != 1:
            init_state = TT.alloc(floatX(0), batch_size, self.n_hids)
        else:
            init_state = TT.alloc(floatX(0), self.n_hids)

I got the following error:

ValueError: When compiling the inner function of scan the following error has been encountered: The initial state (outputs_info in scan nomenclature) of variable IncSubtensor{Set;:int64:}.0 (argument number 4) has dtype float32 and 2 dimension(s), while the result of the inner function for this output has dtype float32 and 2 dimension(s). This could happen if the inner graph of scan results in an upcast or downcast. Please make sure that you use dtypes consistently

I don't think that dtype is the problem, because the TT.alloc (zero vector as init state) has the same dtype, so I'm lost. What's your strategy to debug this kind of problems?
Thanks