Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Images
Hongyu Yan Yadong Mu



Toeplitz Operators on Weighted Bergman Spaces over Tubular Domains
    Lvchang Li College of Mathematics and Informatics,
		South China Agricultural University,
		Guangzhou,
		510640,
		China
		 Email: 20222115006@stu.scau.edu.cn.,	Jiaqing DingCollege of Mathematics and Informatics,
		South China Agricultural University,
		Guangzhou,
		510640,
		China
	Email: 20222115001@stu.scau.edu.cn.,	Haichou Li  Corresponding author,
		College of Mathematics and Informatics,
		South China Agricultural University,
		Guangzhou,
		510640,
		China
		 Email: hcl2016@scau.edu.cn.  Li is supported by NSF of China (Grant No. 12326407 and 12071155). 

    
===========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





  Image-guided object assembly represents a burgeoning research topic in computer vision. This paper introduces a novel task: translating multi-view images of a structural 3D model (for example, one constructed with building blocks drawn from a 3D-object library) into a detailed sequence of assembly instructions executable by a robotic arm. Fed with multi-view images of the target 3D model for replication, the model designed for this task must address several sub-tasks, including recognizing individual components used in constructing the 3D model, estimating the geometric pose of each component, and deducing a feasible assembly order adhering to physical rules. Establishing accurate 2D-3D correspondence between multi-view images and 3D objects is technically challenging. To tackle this, we propose an end-to-end model known as the Neural Assembler. This model learns an object graph where each vertex represents recognized components from the images, and the edges specify the topology of the 3D model, enabling the derivation of an assembly plan. We establish benchmarks for this task and conduct comprehensive empirical evaluations of Neural Assembler and alternative solutions. Our experiments clearly demonstrate the superiority of Neural Assembler.





§ INTRODUCTION



The assembly task necessitates predicting a sequence of operations for the placement of various components. Accurate and efficient assembly algorithms play a pivotal role in robotics. These assembly challenges are pervasive in daily life, as in scenarios like constructing LEGO models <cit.>, assembling furniture <cit.>, and Minecraft <cit.>. In previous research, Chen et al. <cit.> suggested replicating human building order with spatial awareness to construct Minecraft houses without target information. Wang et al. <cit.> introduced a step-by-step approach for assembling LEGO models based on the assembly manual, while the work in <cit.> focused on predicting the 6-DoF pose of each component based on the object's class for assembly. Li et al. <cit.> predicted the poses of parts using a single image.



In this study, we define a new task of image-guided assembly. We are provided with a set of multi-view images captured from a 3-D model, assuming it is built with components from a pre-specified library. The goal of the task is to generate a sequence of fine-grained assembly instructions, encompassing all parameters—such as component types, geometric poses of each component, and assembly order—in accordance with physical rules and suitable for execution by a robotic arm. The task serves as a valuable testbed for advancing vision-guided autonomous systems, presenting a range of technical challenges. Firstly, understanding the correspondence between 2D images and 3D objects is crucial. Given that certain components in the 3D model might be entirely obscured from specific viewpoints, we employ multi-view images (e.g., typically 4 in this study) as input. The algorithm must effectively integrate information from images captured from multiple perspectives. Secondly, estimating critical information for each component is non-trivial. With a 3D library in place, the algorithm needs to segment components across all images and categorize each based on predefined object types in the library, mainly using shape and texture cues. In addition, one also needs estimate the 3-D spatial position and rotation matrix of each component. Thirdly, obtaining typological information for all components is necessary to formulate a physically feasible assembly plan.

Importantly, observations of a component in images are often incomplete, primarily due to frequent occlusions. This poses a substantial challenge in fully understanding and interpreting the scene. Such occlusions are particularly challenging when assembling complex, multi-layerd models. For this novel task, we propose an end-to-end neural network, dubbed as Neural Assembler. The computational pipeline of Neural Assembler is illustrated in Figure <ref>. Taking multi-view images and a 3-D component library as input, Neural Assembler not only identifies each component from images but also determines its 3D pose at each step of assembly. Leveraging images from multiple viewpoints, our method aims to enhance the overall scene understanding and accurately predict the order for placing parts in assembly tasks.

We present two datasets for the proposed image-guided assembly task, namely the CLEVR-Assembly dataset and LEGO-Assembly dataset. Each sample in these datasets comprises images of the object captured from various perspectives, along with the pose of each part (2D keypoints, mask, 3D position, and rotation), and the relationship graph of all components. Comprehensive experiments are conducted on both datasets. Due to the absence of prior work addressing this novel setting like Neural Assembler, we establish two robust baselines for comparison. The evaluations unequivocally demonstrate that Neural Assembler outperforms the baselines across all performance metrics.



§ RELATED WORK

Assembly algorithms
Assembly tasks have utilized computational strategies for part selection and placement, including action scoring <cit.>, genetic algorithms <cit.>, and voxel-based optimization <cit.>. Manual-driven approaches have been investigated <cit.>, with LSTM showing limitations in long sequence predictions <cit.>, and the reinforcement learning method<cit.> struggling with block complexity. Existing research into part-based 3D modeling <cit.> and parts retrieval from 3D meshes <cit.> assumes prior part knowledge. Our novel task of reconstructing 3D models from multi-view images without prior information presents a unique challenge not yet addressed by none of above works.

Multi-view scene understanding
Many scene understanding tasks are intractable in single-view and can only be solved given multi-view inputs.
For example, The SLAM (simultaneous localization and mapping) task requires reconstructing the 3D geometric scene and estimating camera poses given a sequence of video frames. 3D-SIS <cit.> performed instance segmentation by reading a set of RGB-D images. MVPointNet <cit.> was based on a set of RGB-D images and point clouds for instance segmentation. <cit.> performed 3D semantic segmentation and reconstruction based on multiple RGB images. Similarly in our work, the algorithm understands the structure of 3D brick models and predicts the poses of parts based on multi-view images.

Structural modeling for scenes or 3-D models
Recent studies have focused on inferring structural representations like graphs <cit.> and programmatic descriptions <cit.> from images. Techniques range from encoder-decoder architectures for triplet generation <cit.> to efficient graph-based methods <cit.>, CNNs for primitive shape programs <cit.>. An encoder-decoder approach for scene re-rendering <cit.> and transformers for structural change detection <cit.> have also been proposed. Our work differs by using structural representations to predict assembly sequences of object parts.



§ NEURAL ASSEMBLER






 §.§ Task Specification
 

Given several multi-perspective images of a 3-D brick model {I_k}_k=1^K, the corresponding camera parameters with respect to some reference coordinate system, and a predefined 3-D brick (or termed as component, as used exchangeably in this paper) library Lib = {b_1, b_2, …, b_M}, our algorithm identifies the bricks present in the scene, predicts each brick's pose and constructs a relational graph G = {V, E}. The vertex set V correspond to the bricks and the directed edge set E represent spatial configuration that can further be used to derive the assembly instructions. In particular, each brick b_i in the library is denoted as (S_i, T_i) where S_i is assumed to be point clouds and T_i is represented as texture images. In the relationship graph,  each node v_i∈ V describes the i-th brick information v_i=(S_i, T_i, Kps_i, Rot_i, M_i) where Kps_i∈ ([0,1]× [0,1])^K× 2 encodes the 2D keypoints (we use this notation to refer to the planar projection of the brick center) in K views, Rot_i∈  [0, 2π]^K are the rotation angles in K views and M_i∈{0,1}^K× H× W are the binary masks of the brick in K views. The edge e_i,j∈ E explicitly describes the assembly order, where e_i,j=1 only when a brick v_j is placed into the 3-D model after brick v_i.

Here we develop a baseline for the proposed task, namely Neural Assembler, whose computational pipeline is shown in Figure <ref>, Our model leverages a graph convolutional network (GCN) to delineate the assembly sequence effectively. The 3-D poses of objects are inferred from 2D image data across multiple views, exploiting the geometric constraints provided by the camera parameters to ensure spatial consistency in the 3D domain. This framework showcases the capability of capturing complex relational patterns and translating them into a structured assembly protocol.





 §.§ Multi-View Feature Fusion


We adopt the pretrained CLIP image encoder to get the feature maps F^k and CLIP feature vectors v_CLIP^k, k∈{1,2,...K}. Then the features of all views are fused with others, using an adapted implementation of group-based semantic agreement Transformer <cit.>. The scene consensus (denoted as a vector g) of the images from different perspectives are extracted according to the rule: g = 1/K∑_k=1^K Norm(v_CLIP^k),
where Norm(·) is the L_2 normalization. Afterwards, the scene consensus g is dispersed to multi-view image features through channel-wise multiplication:F̂^k=F^k· g,    k=1,2, …, K.


Template as visual prompts. Let the texture image T_i∈ℝ^H_T_i× W_T_i× 3, where (H_T_i, W_T_i) is the texture image size, represent the texture template and point clouds S_i ∈ℝ^N_P× 3 represent the shape template. In all experiments of this work, we fix the parameter N_P to be all 1024, striking a balance between 
compute expense and capability of representing the bricks. For the texture image, we set both H_T_i and W_T_i to 224. A CNN backbone (e.g., ResNet-18 <cit.>) generates template features T_i← CNN(T_i),
and a pointnet backbone (e.g., PointNet <cit.>) generates shape features S_i ← PointNet(S_i).











Transformer decoder. The decoder accepts as input the shape feature library S={S_i}, the texture feature library T={T_i} and the fused image features F̂^k. As shown in Fig. <ref>, the transformer decoder is designed to obtain the image conditioned object query.

The elements from the shape feature library S or the texture feature library T and the N learned object queries O are concatenated to define a query O' = [S, T, O].
Given the fused image features F̂^k and the query O', the library-based object queries f_i^k (i=1,2,…,N, k=1,2, …, K) are obtained through a couple of decoding layers that employ the structure of FS-DETR <cit.>. Then we use an averaging operation to obtain a unified, multi-view feature representation for each object: f_i^global = ∑_k=1^K f_i^k,    i=1,2,…,N.
This approach effectively integrates diverse perspectives to enhance the overall understanding for the object's attribute.

As shown in Fig. <ref>, for the Consensus Dispersion module and Linear Interpolation module, to obtain the object-conditioned image features, we adapt the combination of CLIPSeg <cit.> and group-based semantic agreement Transformer <cit.>. The node features f_i^k serve as conditional prompts. Feature fusion between node features and image features is achieved through linear interpolation. Formally, in each perspective, the object-conditioned image features F_i^k = LI(F̂^k, f_i^k)
where i=1...N,k=1,2,...K, LI is linear interpolation.
Next, the scene consensus g is dispersed to multi-view object-conditioned image features through channel-wise multiplication F̂_i^k = F_i^k· g.



 §.§ Brick Number Prediction

For each view k = 1,2,…,K, we average the N object-conditioned image features F̂_avg^k = 1/N∑_i=1^N F̂_i^k. 
Then the scene feature can be obtained by F̂_scene = 1/K∑_k=1^K F̂_avg^k. Finally a couple of convolution layers are used to predict the number of bricks in the scene.



 §.§ Relation Graph Generation


Predicting the assembly sequence is equivalent to predicting the connections of bricks which can be described using the relationship graph. If brick A is placed on the top of B, then there is a directed edge from B to A. A graph convolutional network <cit.> is adopted to predict the existence of each edge in the graph.
Given the complete graph G = (V, E) with initial node features p_i^0 = f_i^global. Similar to <cit.>, we implemented the GCN using the MLP (multi-layer perception) structure. Each iteration of message passing starts from computing edge features: e_i,j^t+1← MLP([p_i^t, p_j^t]). An average-pooling is performed over all edge features connected to a node, obtaining an updated node feature

    p_i^t+1=1/|{u|(u,i)∈ E}|∑_(u,i)∈ Ee_u,i^t+1.


After gathering the edge features via graph convolutions as {e_i,j^t}_t=1^T, we use another MLP to predict the probability of the existence of each edge, P_(i,j)=MLP(e_i,j^T)


Finally, to determine the assembly sequence during inference, the directed edges are sorted in descending order according to the predicted probability and subsequently incorporated into the directed graph. If a loop is formed after adding an edge, this edge will not be added. The process continues until it reach a state where there exists a vertex that can reach all other vertices in the graph.



 §.§ Pose Estimation




The N object query features {f_i^global}_i=1^N are used for shape and texture classification. The shape prediction head predicts the shape label and the texture prediction head predicts the texture label.

Mask and heatmap prediction. We employ a simple deconvolution layer on the object-conditioned image feature F̂_i^k to obtain the heatmap of keypoint and mask of the object i in each perspective k.










Rotation prediction The rotation angle is represented as a 2D vector representing the sine and cosine values of it. The rotation predictor accepts conditional image features F̂_i^k as input and outputs the sine and cosine value.

Confidence score prediction Since bricks may not be visible at all perspectives, here we predict the the confidence score c_i^k of each brick at each perspective. Specifically, c_i^k represents the Iou(Intersection Over Union) between the predicted mask and ground truth mask.


r0.48
  

  
    < g r a p h i c s >

  

  Illustration of 3D position prediction module.
  


During inference, the pose of each object in 3D space is obtained by merging the poses from each perspective (see Figure <ref>). In more details, for 3D position prediction, our method involves detecting keypoints of the object parts from the view whose confidence score c_i is higher than a threshold θ. Then, utilizing the camera parameters, the rays in 3D space generated by keypoints are used to infer the object's position in 3D space. Each ray R_i is represented as r_i(t) = O_i + t · D_i where O_i is the origin and D_i is the direction.
Our objective is to find a point P that minimizes the function:

    h(Z) = ∑_i^L d(Z, R_i),

where L is the number of rays and d(Z, R_i) represents the shortest distance from the point P to the ray R_i.  Here, the minimization of the objective function h(Z) is approached through the gradient descent method.















 §.§ Training and Loss functions

We train Neural Assembler with full supervision on the generated dataset where each sample we have the groundtruth shape, texture, keypoint, mask, rotation information of each brick, the number of bricks and the relationship graph of bricks. The entire neural network is trained end-to-end with gradient
descent. Our objective function is computed by L=α· L_count + β· L_graph + L_pose, where L_count is the L1 Loss between the predicted number of bricks and ground truth count_gt.

Following <cit.>, bipartite matching is used to find an optimal permutation {σ_i}_i=1^N to match the N object queries and ground truth bricks in the scene. The pose loss of bricks includes the loss of shape, texture, keypoint, mask and rotation.

    L_pose   = L_keypoint + L_mask + γ_1L_rotation
       + γ_2L_shape + γ_3L_texture + γ_4L_confidence,

where L_keypoint is the focal loss <cit.> computed based on the predicted heatmap and ground truth heatmap generated by Kps_σ_i, L_mask is the focal and dice loss between the predicted mask and ground truth mask M_σ_i, L_rotation is the L1 Loss between the prediced sine and cosine and the ground truth sine and cosine of Rot_σ_i, L_shape and L_texture are the cross entropy loss for shape and texture classification and L_confidence is L1 Loss between the predicted confidence score and Iou of the predicted mask and ground truth mask. 
Our model strategically prioritizes the hyperparameters L_keypoint and L_mask due to their critical impact on object detection, essential for accurate object interaction and identification in complex scenes. In contrast, L_rotation, L_shape, L_texture and L_confidence are assigned a reduced weight of 0.1 each, a decision grounded in empirical findings that highlight their relatively minor incremental benefits to overall model efficacy.

L_graph is the loss for relationship graph prediction. Firstly, we define the loss for any subset of the entire edge set. For a subset Ê of the edge set E of the complete graph, the edge loss of Ê is defined as L_Ê=∑_(x,y)∈Ê L_CE(P_(x,y), Ê_(x,y)). Then the L_graph is defined as L_graph=L_E+L_topK_E 

    L_topK_E=1/K_E∑_k=1^K_E L_E_top_k,

where E_top_k is the set of the edges with the top k highest predicted probability. Since the entire relationship graph is a directed graph with sparse edges, the hyperparameter K_E is defined as count_gt + 1.





























§ EXPERIMENTS



Dataset preparation. Experiments are conducted on two self-constructed datasets. The CLEVR-Assembly Dataset, created via the CLEVR-Engine<cit.>, comprises a shape library with 6 brick shapes and 16 textures, a 76.5% visibility probability of each brick per perspective, 7.51 bricks per sample, and an average assembly graph depth of 4.01, with approximately 10K training, 500 validation, and 2000 test samples. The LEGO dataset, synthesized using Pytorch3d, features 12 LEGO brick shapes and 8 textures, an 82.6% visibility probability of each brick per perspective, 7.39 bricks per sample, and an average graph depth of 4.49, also with approximately 10K training, 500 validation, and 2000 test samples. The two datasets, characterized by brick number, occlusion from variable visibility, and complex assembly graph, reflect the complexity of assembly tasks. 

Baseline models. In addressing this novel task, for which no direct baseline exists, we have established a comparative framework against three distinct baseline methods to demonstrate the efficacy of our approach. First, to assess the validity of our assembly order prediction methodology, we introduce a Long Short-Term Memory (LSTM) <cit.> module as a surrogate baseline to contrast with our Graph Convolutional Network (GCN) based module. This comparison aims to highlight the enhanced predictive capabilities our GCN model brings to complex assembly sequences.

Furthermore, for the object pose estimation component, our methodology is rigorously benchmarked against DETR3D <cit.>, a prominent baseline in the realm of object detection within autonomous driving scenarios. This comparison is pivotal in underscoring the adaptability and accuracy of our model in 3D pose estimation, a crucial aspect in varied application domains.

Lastly, in evaluating our multi-view image feature fusion process, we contrast our approach with a method that does not leverage scene consensus. This comparison is instrumental in showcasing the enhanced scene understanding and feature integration our method offers, thus demonstrating its superiority in synthesizing and utilizing multi-view image data.

Implementation details. Our approach is implemented in single-scale version for fair comparison with other works. It incorporates a CLIP<cit.> pre-trained ViT-B/16 image encoder, a PointNet-based <cit.> point cloud encoder, and a ResNet-18<cit.> for texture encoding. We employ a two-layer residual network for brick number prediction. The shape, material, iou prediction heads are implemented using 3-layer MLP and ReLU activations. Rotation prediction also uses a two-layer residual network, and our GCN architecture employs two message-passing layers. Training is conducted on an RTX 3090 GPU using AdamW, with an initial rate of 5e-4, decaying by 0.8 per epoch, a weight decay of 1e-3, and batch size 8 over 10 epochs for both datasets.






Evaluation metrics. In this assembly task, we introduce several metrics to evaluate the performance of our algorithm both at a per-scene and per-step level, providing a holistic measure of our method's efficacy. Specifically, for the per-scene metrics, our approach necessitates the prediction of the entire assembly sequence based on multi-view images, emphasizing the ability to comprehend and reconstruct the complete scene from various perspectives. In contrast, the per-step metrics operate under the assumption that the assembly order is known a priority. Here, we focus on calculating the error between the predicted information for each individual brick and the corresponding ground truth, independent of the assembly order. This allows for a comprehensive evaluation of the method's ability in both holistic scene understanding and step-wise brick analysis.



For per-scene metrics, we evaluate the Complete Rate (completion percentage of the brick model), Order CR (completion rate of the sequence of brick types), Per-scene Acc (accuracy of completely assembling an entire brick model), Count Acc(precision of predicting the number of bricks). For per-step metrics, we evaluate the Pos Acc and Rot Acc (3D position accuracy and rotation accuracy), Shape Acc and Texture Acc (shape accuracy and texture accuracy), Kps Mse (error of the predicted 2D keypoints of the object), mIou (mean Intersection over Union between the predicted mask and the ground truth), the F1-score between predicted relation graph and ground truth relation graph and Per-step Acc(accuracy of correct predictions for each brick's information).





















Results on CLEVR-Assembly. Per-scene quantitative results on the CLEVR-Assembly Dataset are summarized in Table <ref>. Neural Assembler outperforms baseline models in all metrics considered. From Table <ref>, we can see Neural Assembler locates objects more accurately than DETR3D. 

r0.5



!

    < g r a p h i c s >





   The probability distribution of CCA.





 
 
 
 

The metric CCA proposed by <cit.> is adopted here for the brick order evaluation. It denotes the probability distribution of the number of brick that a model can consecutively place from scratch. As shown in Fig.<ref>, LSTM perform worse than GCN. This is because time dependence is not crucial for the assembly order prediction. Instead, the assembly problem requires prediction from complex spatial relationships. The adeptness of GCN in capturing spatial relation plays a critical role in understanding the assembly order.









































Consensus Module As shown in Table <ref>, extracting the consensus can better align the information of the images from various perspectives. Without scene consensus, it is difficult for the model to integrate information from multi-view images to obtain the overall information of each brick.

Number of views Furthermore, we compared the results obtained by accepting  different numbers of images as input. As shown in Tables <ref> and <ref>, the result shows that more perspectives as input can improve the performance. This is because each brick may not be seen from some perspectives due to the existence of occlusion. More views mean more information for prediction.





Fig. <ref> further shows the generated assembly instructions for a brick model in the CLEVR-Assembly Dataset. Perspectives with confidence score greater than 0.66 are selected to infer the brick's information. It is evident that Neural Assembler is adept at excluding perspectives where bricks are obscured by predicting confidence scores, thereby identifying optimal perspectives for predicting positional information. Concurrently, it is capable of predicting the structure between bricks to determine the appropriate assembly sequence.



Results on LEGO-Assembly. Different from CLEVR-Dataset, LEGO bricks are connected through slots. It is easier to infer the position of the bricks based on the connection constraints between the LEGO bricks, as long as the rough position of the LEGO bricks is predicted. However, there are many challenges in predicting the assembly sequence of LEGO brick models. For instance, the more compact assembly of LEGO bricks results in increased occlusion.

The LEGO brick will only have rotations chosen from
(0^∘, 90^∘ , 180^∘, 270^∘). Meanwhile, the position of LEGO bricks is discretized. We adopt the connection-constrained inference subroutine and an inference-by-synthesis subroutine used in <cit.> to predict the position and rotation angle for each view, and then integrated them through voting. The results in Table <ref> and Table <ref> shows that Neural Assembler can yield more
accurate results than other baselines.
Fig.<ref> further shows the generated assembly instructions for a LEGO model.






Real-world experiments. 

To confirm the model's generalizability, a comprehensive test dataset is constructed, complete with annotations for each brick’s shape, position, and rotation. For each sample, we acquired real brick images using a Realsense camera in the real world and generated corresponding simulated images in the simulation environment employing real-world camera parameters to ensure the consistent coordinate between the simulated and real environments. The dataset encompasses 5 brick types and 7 textures, averaging 6.1 bricks per brick model.

















To evaluate the Neural Assembler, we collected point clouds and textures from real bricks. This data facilitated the creation of a synthetic dataset, used for fine-tuning the model initially trained on the CLEVR-Assembly dataset.

As indicated in Table <ref>, the Neural Assembler achieves performance in real-world experiments close to the results obtained in simulated environments, demonstrating its robust applicability. Fig. <ref> presents the result on the real world dataset.










Discussion. As shown in Fig. <ref>, the occlusion still greatly affects the performance of the model, especially the objects under the brick model will be greatly blocked by the bricks pressing on it. To alleviate this problem, in future work, we plan to enhance model performance with a deeper integration of physical scene understanding. The model is expected to not only interpret the visual aspects but also the underlying physical principles governing the scene. 





§ CONCLUSION

We study the problem of generating robotic assembly instructions from multi-view images and propose Neural Assembler, a model that predicts the assembly instructions of the brick model. The key idea behind our model is to learn the graph structure to predict the relationships among bricks and infer the 3D pose according to multi-view images. Results show that our model outperforms existing methods on the newly collected CLEVR-Assembly and LEGO-Assembly Dataset.





































































ieeenat_fullname
























§ APPENDIX / SUPPLEMENTAL MATERIAL



The data generation pipeline for CLEVR-Assembly and LEGO-Assembly dataset is introduced in Section <ref>. Then we provide more implementation details and hyperparameters of models in Section <ref>. Section <ref> shows the details of the real-world robotic experiment. 



 §.§ Dataset Generation


CLEVR-Assembly Dataset In constructing the CLEVR-Assembly dataset, each sample is randomly generated. For each step of assembly, a brick with a random shape and texture is selected, followed by randomizing its horizontal position (x, y) and rotation angle. The operation is rolled back if the brick is unstable upon free fall. Each brick's information including keypoints, mask, 3D coordinate, rotation, shape, and texture is recorded, and relationships with other bricks are computed. All models are confined within a [-3, 3]×[-3, 3] horizontal area, with the smallest cubic brick being 1×1×1 in size. 

The CLEVR-Assembly dataset comprises various brick shapes: cubes (1×1×1), rectangular prisms (1×1×2, 1×2×0.5, 2×2×0.5), cylinder with a base diameter of 1 unit and a height of 1 unit, and triangular prisms featuring a square base with each side measuring √(2) units and a height of √(2)/2 units. For textures, 8 distinct brick textures are sourced, and their average colors are used, leading to 16 unique textures. This results in a total of 96 different brick categories.

LEGO-Assembly Dataset 
LEGO-Assembly Dataset For the LEGO-Assembly dataset construction, each sample begins with selecting a base brick randomly. Subsequently, at every step, we choose a random brick and determine feasible poses for this brick, considering the connection constraints of the pre-assembled brick model. A pose is then randomly selected from this set as the brick's final placement.

The LEGO-Assembly dataset selects 12 distinct brick types from the LEGO library and utilizes 8 different colors for textures, resulting in a total of 96 unique brick categories.

Four cameras are positioned at the left-front, left-back, right-front, and right-back of the brick model. Their translations are randomly chosen within a sphere of radius 1.5 at a distance of 12 from the origin. The rotation is sampled from (0, (90· k ± 30 )^∘, (45 ± 15)^∘) for camera k.



 §.§ Implementation Details


Hyperparameters For training loss:

    L    = α· L_count + β· L_graph + L_pose,
    
    L_pose   = L_keypoint + L_mask + γ_1L_rotation
       + γ_2L_shape + γ_3L_texture + γ_4L_confidence,

We set α to 0.2, β to 0.5, γ_1 to 0.1, γ_2 to 0.1, γ_3 to 0.1 and γ_4 to 0.1.
All models are trained using AdamW, with an initial rate of 5e-4, decaying by 0.8 per epoch, a weight decay of 1e-3, and batch size 8 over 10 epochs for both datasets. We use the pre-trained ViT-B/16 weights and fine-tune it with the learning rate setting to the same value as other modules.

Details of 3D Pose Inference  The center of each brick is defined as the keypoint. For 3D pose estimation, we select the perspective with a confidence score greater than a threshold and extract its 2D keypoint coordinates 
(x,y). In the camera coordinate system, we derive the world coordinates for points (x,y,-1) and (x,y,1) in the camera coordinate, which form the predicted ray for the keypoint from the perspective. Consequently, the 3D position of the keypoint is determined by the intersection of rays predicted from multiple perspectives.

Model Architecture 
A pre-trained Vision Transformer (ViT-B/16) processes an image of size 224× 224, yielding image features of dimension 768×(196+1). These features are then transformed via a fully connected layer into a feature space of 256×(196+1), where 196 represents the number of tokens, equating to 14×14, and the resulting CLIP feature vector has a dimension of 1×256. Concurrently, the PointNet processes the point cloud of size N_1×1024×3 to extract N_1×256 shape features of the brick. For texture information, N_2 images of size 224×224, processed through the ResNet-18 network, yield the texture features of dimension N_2×256.

For the Transformer Decoder, the query is constituted by object queries of dimension (N_1+N_2+16)×256. The key is established as 196×256 image embeddings, augmented with positional encoding, while the value is set as the image embeddings, obtaining the global object features. The per-view object feature and the fused image feature  are intricately processed by  the Linear Interpolation module to yield the object-conditioned image feature F̂_̂î^̂k̂ of dimension 256×14×14.



 §.§ The computational complexity

The runtime of the baseline consists of forward propagation (T1)
and inferring bricks’ poses and assembly sequence (T2). The
average T1 for Neural Assembler/ DETR3D / LSTM is 0.24s
/ 0.23s / 0.20s respectively. For T2, the averages are 1.79s /
1.04s / 1.32s for Neural Assembler / DETR3D / LSTM respectively. The total runtime T1 + T2 is acceptable.





 §.§ Real-World Robotic Experiment



To evaluate the practical applicability of Neural Assembler, a real-world assembly experiment is conducted. The experimental setup comprises an Aubo-i5 robotic arm equipped with an Intel RealSense D435i RGB-D camera, facilitating precise visual perception. The manipulation component involves a Robotiq 2F-85 two-finger gripper, providing adept grasping capabilities.

The grasping process begins as the robotic arm captures images from four distinct perspectives within the Detection Area, as illustrated in Fig. <ref>. This enables the extraction of structural information about the brick model. Subsequently, the arm relocates to the Assembly Area, where it utilizes bricks from the Brick Library to reconstruct the brick model. At each step of assembly, the arm determines the brick's location within the library based on its shape and texture. The brick is then grasped from a vertical direction and positioned at the predicted pose. 


