Dfyjkt

I am trying to create an English to French translator. I have a basic model which works fairly well:

Average step time: 232.3

Final loss: 0.4969

Model:

Layer (type) Output Shape Param # 
=================================================================
embedding_1 (Embedding) (None, 15, 336) 67200 
_________________________________________________________________
lstm_1 (LSTM) (None, 256) 607232 
_________________________________________________________________
repeat_vector_1 (RepeatVecto (None, 21, 256) 0 
_________________________________________________________________
dropout_1 (Dropout) (None, 21, 256) 0 
_________________________________________________________________
lstm_2 (LSTM) (None, 21, 256) 525312 
_________________________________________________________________
dropout_2 (Dropout) (None, 21, 256) 0 
_________________________________________________________________
time_distributed_1 (TimeDist (None, 21, 336) 86352 
=================================================================
Total params: 1,286,096
Trainable params: 1,286,096
Non-trainable params: 0
_________________________________________________________________

Python:

model = Sequential()
model.add(Embedding(en_vocab_size, fr_vocab_size, input_length=en_max_len, mask_zero=True))
model.add(LSTM(256))
model.add(RepeatVector(fr_max_len))
model.add(Dropout(0.5))
model.add(LSTM(256, return_sequences=True))
model.add(Dropout(0.2))
model.add(TimeDistributed(Dense(fr_vocab_size, activation='softmax')))

I then tried to implement another LSTM layer and two 1d convolutional layers:

Average epoch time: 402s

Final loss: 1.0899

Layer (type) Output Shape Param #
=================================================================
embedding_1 (Embedding) (None, 15, 336) 67200
_________________________________________________________________
dropout_1 (Dropout) (None, 15, 336) 0
_________________________________________________________________
conv1d_1 (Conv1D) (None, 15, 32) 32288
_________________________________________________________________
max_pooling1d_1 (MaxPooling1 (None, 7, 32) 0
_________________________________________________________________
dropout_2 (Dropout) (None, 7, 32) 0
_________________________________________________________________
conv1d_2 (Conv1D) (None, 7, 16) 2064
_________________________________________________________________
max_pooling1d_2 (MaxPooling1 (None, 3, 16) 0
_________________________________________________________________
dropout_3 (Dropout) (None, 3, 16) 0
_________________________________________________________________
lstm_1 (LSTM) (None, 128) 74240
_________________________________________________________________
repeat_vector_1 (RepeatVecto (None, 21, 128) 0
_________________________________________________________________
dropout_4 (Dropout) (None, 21, 128) 0
_________________________________________________________________
lstm_2 (LSTM) (None, 21, 512) 1312768
_________________________________________________________________
dropout_5 (Dropout) (None, 21, 512) 0
_________________________________________________________________
lstm_3 (LSTM) (None, 21, 128) 328192
_________________________________________________________________
dropout_6 (Dropout) (None, 21, 128) 0
_________________________________________________________________
time_distributed_1 (TimeDist (None, 21, 336) 43344
=================================================================
Total params: 1,860,096
Trainable params: 1,860,096
Non-trainable params: 0
_________________________________________________________________

Python:

model = Sequential()
model.add(Embedding(en_vocab_size, fr_vocab_size, input_length=en_max_len))
model.add(Dropout(0.2))
model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(Dropout(0.3))
model.add(Conv1D(filters=16, kernel_size=4, padding='same', activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(Dropout(0.2))
model.add(LSTM(128))
model.add(RepeatVector(fr_max_len))
model.add(Dropout(0.2))
model.add(LSTM(256, return_sequences=True))
model.add(Dropout(0.5))
model.add(LSTM(256, return_sequences=True))
model.add(Dropout(0.2))
model.add(TimeDistributed(Dense(fr_vocab_size, activation='softmax')))

You can see that the second one not only took longer to train, had a larger loss but also was far less accurate. Why would this be? My assumption is that I implemented the convolutional layers incorrectly. What is the best way to implement a convolutional layer in a recurrent neural network (or lstm network)?