title: How to serve a tensorflow-module, specifically Universal Sentence Encoder?
body: <p>I have spent several hours trying to set up Tensorflow serving of the Tensorflow-hub module, "Universal Sentence Encoder." There is a similar question here:</p>
  
  <p><a href="https://stackoverflow.com/questions/50788080/how-to-make-the-tensorflow-hub-embeddings-servable-using-tensorflow-serving">How to make the tensorflow hub embeddings servable using tensorflow serving?</a> </p>
  
  <p>I have been doing this on a Windows machine.</p>
  
  <p>This is the code I used to build the model:</p>
  
  <pre><code>import tensorflow as tf
  import tensorflow_hub as hub
  
  MODEL_NAME = 'test'
  VERSION = 1
  SERVE_PATH = './models/{}/{}'.format(MODEL_NAME, VERSION)
  
  with tf.Graph().as_default():
    module = hub.Module("https://tfhub.dev/google/universal-sentence- 
    encoder/1")
    text = tf.placeholder(tf.string, [None])
    embedding = module(text)
  
    init_op = tf.group([tf.global_variables_initializer(), 
    tf.tables_initializer()])
    with tf.Session() as session:
    session.run(init_op)
      tf.saved_model.simple_save(
      session,
      SERVE_PATH,
      inputs = {"text": text},
      outputs = {"embedding": embedding},
      legacy_init_op = tf.tables_initializer()        
     )
  </code></pre>
  
  <p>I have gotten to the point where running the following line:</p>
  
  <pre><code>saved_model_cli show --dir ${PWD}/models/test/1 --tag_set serve --signature_def serving_default
  </code></pre>
  
  <p>gives me the following result:</p>
  
  <pre><code>The given SavedModel SignatureDef contains the following input(s):
  inputs['text'] tensor_info:
    dtype: DT_STRING
    shape: (-1)
    name: Placeholder:0
  The given SavedModel SignatureDef contains the following output(s):
   outputs['embedding'] tensor_info:
    dtype: DT_FLOAT
    shape: (-1, 512)
    name: module_apply_default/Encoder_en/hidden_layers/l2_normalize:0
  </code></pre>
  
  <p>I have then tried running:</p>
  
  <pre><code>saved_model_cli run --dir ${PWD}/models/test/1 --tag_set serve --signature_def serving_default --input_exprs 'text=["what this is"]'
  </code></pre>
  
  <p>which gives the error:</p>
  
  <pre><code>  File "&lt;string&gt;", line 1
  [what this is]
           ^
  SyntaxError: invalid syntax
  </code></pre>
  
  <p>I've tried changing the format of the part 'text=["what this is"]', but nothing worked for me.</p>
  
  <p>Regardless of if this part works, the main goal is to set up the module for serving and create a callable API. </p>
  
  <p>I've tried with docker, the following line:</p>
  
  <pre><code>docker run -p 8501:8501 --name tf-serve -v ${PWD}/models/:/models -t tensorflow/serving --model_base_path=/models/test
  </code></pre>
  
  <p>Things appear to be set up properly: </p>
  
  <pre><code>Building single TensorFlow model file config:  model_name: model model_base_path: /models/test
  2018-10-09 07:05:08.692140: I tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.
  2018-10-09 07:05:08.692301: I tensorflow_serving/model_servers/server_core.cc:517]  (Re-)adding model: model
  2018-10-09 07:05:08.798733: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: model version: 1}
  2018-10-09 07:05:08.798841: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: model version: 1}
  2018-10-09 07:05:08.798870: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: model version: 1}
  2018-10-09 07:05:08.798904: I external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:360] Attempting to load native SavedModelBundle in bundle-shim from: /models/test/1
  2018-10-09 07:05:08.798947: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /models/test/1
  2018-10-09 07:05:09.055822: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
  2018-10-09 07:05:09.338142: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
  2018-10-09 07:05:09.576751: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:162] Restoring SavedModel bundle.
  2018-10-09 07:05:28.975611: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:138] Running MainOp with key saved_model_main_op on SavedModel bundle.
  2018-10-09 07:06:30.941577: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:259] SavedModel load for tags { serve }; Status: success. Took 82120946 microseconds.
  2018-10-09 07:06:30.990252: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:83] No warmup data file found at /models/test/1/assets.extra/tf_serving_warmup_requests
  2018-10-09 07:06:31.046262: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: model version: 1}
  2018-10-09 07:06:31.184541: I tensorflow_serving/model_servers/server.cc:285] Running gRPC ModelServer at 0.0.0.0:8500 ...
  [warn] getaddrinfo: address family for nodename not supported
  2018-10-09 07:06:31.221644: I tensorflow_serving/model_servers/server.cc:301] Exporting HTTP/REST API at:localhost:8501 ...
  [evhttp_server.cc : 235] RAW: Entering the event loop ...
  </code></pre>
  
  <p>I've tried </p>
  
  <pre><code>curl http://localhost:8501/v1/models/test
  </code></pre>
  
  <p>which gives</p>
  
  <pre><code>{ "error": "Malformed request: GET /v1/models/test:predict" }
  </code></pre>
  
  <p>and </p>
  
  <pre><code>curl -d '{"text": "Hello"}' -X POST http://localhost:8501/v1/models/test:predict
  </code></pre>
  
  <p>which gives </p>
  
  <pre><code>{ "error": "JSON Parse error: Invalid value. at offset: 0" }
  </code></pre>
  
  <p>A similar question is here </p>
  
  <p><a href="https://stackoverflow.com/questions/52671138/tensorflow-serving-rest-api-returns-malformed-request-error">Tensorflow Serving: Rest API returns &quot;Malformed request&quot; error</a></p>
  
  <p>Just looking for any way to get this module serving. Thanks.</p>