## Approach Figure 1: Overview of Autoregressive Modeling. (top) We use a VQ-VAE to extract a low-dimensional discrete representation of 3D shapes. Using a patch-wise encoder enables independently encoding local context and allows downstream tasks with partial observations. (bottom) We learn a transformer-based autoregressive model over the latent representation. Using randomized sampling orders in allows learning a ‘non-sequential’ autoregressive shape prior that can condition on arbitrary set of partial latent observations.

# Discretized Latent Space for 3D Shapes

To learn an effective autoregressive model, we aim to reduce the high-dimensional continuous 3D shape representation to a lower-dimensional discrete latent space. Towards this, we adapt the VQ-VAE  framework and learn a 3D-VQ-VAE whose encoder Eψ can compute the desired low-dimensional representation, and the decoder Dψ can map this latent space back to/ 3D shapes. Given a 3D shape X with spatial dimension of D3 we have

where VQ is the Vector Quantization step that maps a vector to the nearest element in the codebook. The encoder Eψ has a very large receptive field. Unfortunately, this is not a desirable property for tasks such as shape completion since the latent codes for encoded partial shapes may differ significantly from those of the encoded full shape — thus partial observations of shape may not correspond to partial observations of latent variables. To overcome this challenge, we propose Patch-wise Encoding VQ-VAE or P-VQ-VAE that encodes the local shape regions independently, while decoding them jointly — this allows the discrete encodings to only depend on local context.

# Non-sequential Autoregressive Modeling

The latent space Z is a 3D grid of tokens representing the original 3D shape. We can thus reduce the task of learning the distribution over continuous 3D shapes to learning p(Z). An autoregressive model like transformer can model this distribution by factoring it as a product of location specific conditionals. However, this factorization assumes a fixed ordering in which the tokens are observed/generated. This is generally not true for conditioning tasks like shape completion, reconstruction etc. We follow the observation from  that the joint distribution p(Z) can be factorized into terms of the form p(zi| O), where O is a random set of observed variables. Hence we use a randomly permuted sequence of latent variables {zg1, zg2, zg3, …} for autoregressively modeling the distribution p(z):

The non-sequential autoregressive network models the distribution over the latent variables Z, which can be mapped to full 3D shapes X = Dψ(Z)

# Conditional Generation

• Shape Completion: The proposed P-VQ-VAE encodes local regions independently. This enables us to map partially observed shape Xp to corresponding observed latent variables O = {zg1, zg2, …., zgk}. In particular, we can formulate the task of shape completion as:
{"ft_lang_label":"__label__en","ft_lang_prob":0.85663736,"math_prob":0.92275995,"size":3685,"snap":"2023-40-2023-50","text_gpt3_token_len":798,"char_repetition_ratio":0.13012768,"word_repetition_ratio":0.0,"special_character_ratio":0.1945726,"punctuation_ratio":0.096423015,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.98081416,"pos_list":[0,1,2],"im_url_duplicate_count":[null,1,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2023-09-26T15:05:16Z\",\"WARC-Record-ID\":\"<urn:uuid:6f3eff25-62f8-4a45-aad9-bf7f40957e90>\",\"Content-Length\":\"94223\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:095eb77b-3ba9-41a2-9bce-12c83d3cce53>\",\"WARC-Concurrent-To\":\"<urn:uuid:ba3f1b31-8248-42e0-8e4c-d21b78e8b221>\",\"WARC-IP-Address\":\"128.2.43.152\",\"WARC-Target-URI\":\"https://mscvprojects.ri.cmu.edu/2021teamb/category/technical-report/\",\"WARC-Payload-Digest\":\"sha1:VZ5HH6Q5IIS5H6M6DO4XP2HYQSFKCXH6\",\"WARC-Block-Digest\":\"sha1:IHUH56JRIQNV7OWXIPTNFQS2CIOAEGA5\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2023/CC-MAIN-2023-40/CC-MAIN-2023-40_segments_1695233510214.81_warc_CC-MAIN-20230926143354-20230926173354-00401.warc.gz\"}"}