
# DALL-E Mini Running in the Browser (work in progress)


### Notes:
 * Working tflite conversion in [this notebook](https://colab.research.google.com/gist/josephrocca/f427377f76c574f1c1e8e4d6d62c34b6/tflite-dalle-mini-conversion-separated-encoder-and-decoder.ipynb).
   * Note that the encoder and decoder need to be converted separately for some reason. [More info on this bug](https://github.com/tensorflow/tensorflow/issues/56629#issuecomment-1172928973).
   * But these models currently require TF Select operators due to bitwise operations that aren't supported in the default tflite opset. [This issue](https://github.com/tensorflow/tfjs/issues/5844) tracks the potential inclusion of TF Select ops in tfjs-tflite, so models like this can be used on the web.
   * The encoder requires the TF Select `Erf` op. The decoder requires these TF Select ops: `Bitcast`, `BitwiseOr`, `BitwiseXor`, `ConcatV2`, `Erf`, `LeftShift`, `RightShift`, `ScatterNd`, `SelectV2`, `StridedSlice`
   * Here are the two models:
     * [dalle_bart_encoder_flax_model](https://huggingface.co/rocca/dalle-mini-js/blob/main/debug/dalle_bart_encoder_flax_model.tflite)
     * [dalle_bart_decoder_flax_model](https://huggingface.co/rocca/dalle-mini-js/blob/main/debug/dalle_bart_decoder_flax_model.tflite)
 * ONNX conversion is blocked on support for `Bitcast`, `BitwiseOr` and `BitwiseXor`: https://github.com/onnx/tensorflow-onnx/issues/1985 The need for the `BitwiseAnd` op can be avoided by pulling `jax.random.PRNGKey` outside of the encode-decode function, but I'm not sure where the other ops are coming from - netron doesn't seem to "expand" certain types of nodes, so I think they're hidden within some other nodes.
 * TF.js conversion is blocked on lack of support for these ops: `BitwiseXor`, `Bitcast`, `LeftShift`, `BitwiseOr`, `RightShift` ([relevant issue](https://github.com/tensorflow/tfjs/issues/6599))

{"license": "mit"}
tflite license:mit region:us
TAGS
#tflite #license-mit #region-us 


# DALL-E Mini Running in the Browser (work in progress)


### Notes:
 * Working tflite conversion in this notebook.
   * Note that the encoder and decoder need to be converted separately for some reason. More info on this bug.
   * But these models currently require TF Select operators due to bitwise operations that aren't supported in the default tflite opset. This issue tracks the potential inclusion of TF Select ops in tfjs-tflite, so models like this can be used on the web.
   * The encoder requires the TF Select 'Erf' op. The decoder requires these TF Select ops: 'Bitcast', 'BitwiseOr', 'BitwiseXor', 'ConcatV2', 'Erf', 'LeftShift', 'RightShift', 'ScatterNd', 'SelectV2', 'StridedSlice'
   * Here are the two models:
     * dalle_bart_encoder_flax_model
     * dalle_bart_decoder_flax_model
 * ONNX conversion is blocked on support for 'Bitcast', 'BitwiseOr' and 'BitwiseXor': URL The need for the 'BitwiseAnd' op can be avoided by pulling 'URL.PRNGKey' outside of the encode-decode function, but I'm not sure where the other ops are coming from - netron doesn't seem to "expand" certain types of nodes, so I think they're hidden within some other nodes.
 * URL conversion is blocked on lack of support for these ops: 'BitwiseXor', 'Bitcast', 'LeftShift', 'BitwiseOr', 'RightShift' (relevant issue)

# DALL-E Mini Running in the Browser (work in progress) ### Notes:
 * Working tflite conversion in this notebook.
   * Note that the encoder and decoder need to be converted separately for some reason. More info on this bug.
   * But these models currently require TF Select operators due to bitwise operations that aren't supported in the default tflite opset. This issue tracks the potential inclusion of TF Select ops in tfjs-tflite, so models like this can be used on the web.
   * The encoder requires the TF Select 'Erf' op. The decoder requires these TF Select ops: 'Bitcast', 'BitwiseOr', 'BitwiseXor', 'ConcatV2', 'Erf', 'LeftShift', 'RightShift', 'ScatterNd', 'SelectV2', 'StridedSlice'
   * Here are the two models:
     * dalle_bart_encoder_flax_model
     * dalle_bart_decoder_flax_model
 * ONNX conversion is blocked on support for 'Bitcast', 'BitwiseOr' and 'BitwiseXor': URL The need for the 'BitwiseAnd' op can be avoided by pulling 'URL.PRNGKey' outside of the encode-decode function, but I'm not sure where the other ops are coming from - netron doesn't seem to "expand" certain types of nodes, so I think they're hidden within some other nodes.
 * URL conversion is blocked on lack of support for these ops: 'BitwiseXor', 'Bitcast', 'LeftShift', 'BitwiseOr', 'RightShift' (relevant issue)
TAGS
#tflite #license-mit #region-us 
 # DALL-E Mini Running in the Browser (work in progress) ### Notes:
 * Working tflite conversion in this notebook.
   * Note that the encoder and decoder need to be converted separately for some reason. More info on this bug.
   * But these models currently require TF Select operators due to bitwise operations that aren't supported in the default tflite opset. This issue tracks the potential inclusion of TF Select ops in tfjs-tflite, so models like this can be used on the web.
   * The encoder requires the TF Select 'Erf' op. The decoder requires these TF Select ops: 'Bitcast', 'BitwiseOr', 'BitwiseXor', 'ConcatV2', 'Erf', 'LeftShift', 'RightShift', 'ScatterNd', 'SelectV2', 'StridedSlice'
   * Here are the two models:
     * dalle_bart_encoder_flax_model
     * dalle_bart_decoder_flax_model
 * ONNX conversion is blocked on support for 'Bitcast', 'BitwiseOr' and 'BitwiseXor': URL The need for the 'BitwiseAnd' op can be avoided by pulling 'URL.PRNGKey' outside of the encode-decode function, but I'm not sure where the other ops are coming from - netron doesn't seem to "expand" certain types of nodes, so I think they're hidden within some other nodes.
 * URL conversion is blocked on lack of support for these ops: 'BitwiseXor', 'Bitcast', 'LeftShift', 'BitwiseOr', 'RightShift' (relevant issue)
passage: TAGS
#tflite #license-mit #region-us 
# DALL-E Mini Running in the Browser (work in progress)### Notes:
 * Working tflite conversion in this notebook.
   * Note that the encoder and decoder need to be converted separately for some reason. More info on this bug.
   * But these models currently require TF Select operators due to bitwise operations that aren't supported in the default tflite opset. This issue tracks the potential inclusion of TF Select ops in tfjs-tflite, so models like this can be used on the web.
   * The encoder requires the TF Select 'Erf' op. The decoder requires these TF Select ops: 'Bitcast', 'BitwiseOr', 'BitwiseXor', 'ConcatV2', 'Erf', 'LeftShift', 'RightShift', 'ScatterNd', 'SelectV2', 'StridedSlice'
   * Here are the two models:
     * dalle_bart_encoder_flax_model
     * dalle_bart_decoder_flax_model
 * ONNX conversion is blocked on support for 'Bitcast', 'BitwiseOr' and 'BitwiseXor': URL The need for the 'BitwiseAnd' op can be avoided by pulling 'URL.PRNGKey' outside of the encode-decode function, but I'm not sure where the other ops are coming from - netron doesn't seem to "expand" certain types of nodes, so I think they're hidden within some other nodes.
 * URL conversion is blocked on lack of support for these ops: 'BitwiseXor', 'Bitcast', 'LeftShift', 'BitwiseOr', 'RightShift' (relevant issue)