Dfyjkt

Trying to serve a Tensorflow tensor2tensor Chinese to English model and am having trouble querying my model. This is the client error I am receiving:

(test) root@ubuntu-c-8-16gib-sfo2-01:~/T2T_Model# t2t-query-server --server=0.0.0.0:9000 --servable_name=transformer --problem=translate_enzh_wmt32k_rev --data_dir=/root/T2T_Model/t2t_data --inputs_once='Hello my name is John.'

Traceback (most recent call last):
 File "/usr/local/bin/t2t-query-server", line 17, in <module>
 tf.app.run()
 File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 125, in run
 _sys.exit(main(argv))
 File "/usr/local/bin/t2t-query-server", line 12, in main
 query.main(argv)
 File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/serving/query.py", line 89, in main
 outputs = serving_utils.predict([inputs], problem, request_fn)
 File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/serving/serving_utils.py", line 157, in predict
 predictions = request_fn(examples)
 File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/serving/serving_utils.py", line 113, in _make_grpc_request
 response = stub.Predict(request, timeout_secs)
 File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 533, in __call__
 return _end_unary_response_blocking(state, call, False, None)
 File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
 raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
 status = StatusCode.INVALID_ARGUMENT
 details = "Requested more than 0 entries, but params is empty. Params shape: [1,4,8,0,64]
 [[node transformer/while/GatherNd_32 = GatherNd[Tindices=DT_INT32, Tparams=DT_FLOAT, _output_shapes=[[?,8,?,?,64]], _device="/job:localhost/replica:0/task:0/device:CPU:0"](transformer/while/Reshape_65, transformer/while/stack)]]"
 debug_error_string = ""created":"@1542086942.107507941","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"Requested more than 0 entries, but params is empty. Params shape: [1,4,8,0,64]nt [[node transformer/while/GatherNd_32 = GatherNd[Tindices=DT_INT32, Tparams=DT_FLOAT, _output_shapes=[[?,8,?,?,64]], _device="/job:localhost/replica:0/task:0/device:CPU:0"](transformer/while/Reshape_65, transformer/while/stack)]]","grpc_status":3"
>

The model server seems to be working fine and responding with the same error:

(test) root@ubuntu-c-8-16gib-sfo2-01:~/T2T_Model# tensorflow_model_server --port=9000 --model_name=transformer --model_base_path=/root/T2T_Model/t2t_train/translate_enzh_wmt32k/transformer-transformer_base/export
2018-11-13 05:28:29.116290: I tensorflow_serving/model_servers/server.cc:82] Building single TensorFlow model file config: model_name: transformer model_base_path: /root/T2T_Model/t2t_train/translate_enzh_wmt32k/transformer-transformer_base/export
2018-11-13 05:28:29.116412: I tensorflow_serving/model_servers/server_core.cc:461] Adding/updating models.
2018-11-13 05:28:29.116424: I tensorflow_serving/model_servers/server_core.cc:558] (Re-)adding model: transformer
2018-11-13 05:28:29.216782: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable name: transformer version: 1542073770
2018-11-13 05:28:29.216806: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version name: transformer version: 1542073770
2018-11-13 05:28:29.216815: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version name: transformer version: 1542073770
2018-11-13 05:28:29.216830: I external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:363] Attempting to load native SavedModelBundle in bundle-shim from: /root/T2T_Model/t2t_train/translate_enzh_wmt32k/transformer-transformer_base/export/1542073770
2018-11-13 05:28:29.216838: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /root/T2T_Model/t2t_train/translate_enzh_wmt32k/transformer-transformer_base/export/1542073770
2018-11-13 05:28:29.537966: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags serve 
2018-11-13 05:28:29.597214: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2018-11-13 05:28:29.722289: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:162] Restoring SavedModel bundle.
2018-11-13 05:28:30.139345: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:138] Running MainOp with key saved_model_main_op on SavedModel bundle.
2018-11-13 05:28:30.227063: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:259] SavedModel load for tags serve ; Status: success. Took 1010210 microseconds.
2018-11-13 05:28:30.227116: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:83] No warmup data file found at /root/T2T_Model/t2t_train/translate_enzh_wmt32k/transformer-transformer_base/export/1542073770/assets.extra/tf_serving_warmup_requests
2018-11-13 05:28:30.227223: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version name: transformer version: 1542073770
2018-11-13 05:28:30.229398: I tensorflow_serving/model_servers/server.cc:286] Running gRPC ModelServer at 0.0.0.0:9000 ...

2018-11-13 05:59:38.052592: W external/org_tensorflow/tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: Requested more than 0 entries, but params is empty. Params shape: [1,4,8,0,64]

Would appreciate any comments or suggestions as to what is wrong here.