You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
execuse me, in the paper "Layer Normalization,Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton", it said that the mean and variance is computed over all the hidden units in the same layer, and different training cases have different normalization terms. So I think the mean should be computed like this:
I tried this, but the error message show below: ValueError: Dimensions must be equal, but are 128 and 32 for 'Embedding_Encoder_Layer/Encoder_Residual_Block/encoder_block_0/layer_norm_0/sub' (op: 'Sub') with input shapes: [32,?,1,128], [32].
I guess there are many places have to adjust to fit it XD
@alznn maybe you can try tf.nn.moments(inputs, axes). I use this, and no error. And you need to pay attention that after compute the mean and variance, you need to use the beta and gamma to renormalize the distribution, the shape of beta and gamma is inputs.shape[-1].
Thanks for your question! You are right in that right now it looks more like an instance normalization rather than a layer normalization. If you check this line I actually attempted using both normalization methods and they didn't show much difference so I sticked with one!
Activity
alznn commentedon Oct 8, 2018
I tried this, but the error message show below:
ValueError: Dimensions must be equal, but are 128 and 32 for 'Embedding_Encoder_Layer/Encoder_Residual_Block/encoder_block_0/layer_norm_0/sub' (op: 'Sub') with input shapes: [32,?,1,128], [32].
I guess there are many places have to adjust to fit it XD
PanXiebit commentedon Oct 8, 2018
@alznn maybe you can try
tf.nn.moments(inputs, axes). I use this, and no error. And you need to pay attention that after compute the mean and variance, you need to use the beta and gamma to renormalize the distribution, the shape ofbetaandgammaisinputs.shape[-1].localminimum commentedon Oct 26, 2018
Thanks for your question! You are right in that right now it looks more like an instance normalization rather than a layer normalization. If you check this line I actually attempted using both normalization methods and they didn't show much difference so I sticked with one!