Recent advances in artificial intelligence (AI) and machine learning (ML) have given rise to an immense popularity of inverse design, and the field shows little signs of slowing down. In inverse design, desired properties are specified a priori, and such methods generate compounds fitting that description. Significant progress has been made in this area and a plethora of approaches for deep learning in molecular design has been published the last few years. Many methods include reinforcement learning to generate molecules, most often in the form of SMILES strings. Other popular methods include generative methods such as recursive neural networks, generative adversarial networks or variational autoencoders, which are sometimes steered with reinforcement learning to control the molecular properties. The SMILES format in itself is nothing but amazing. Using SMILES is also convenient for the AI algorithm, since a string is trivial to manipulate and transform. In addition, there are success stories of using SMILES in the area of generative design. However, all molecules are 3D objects and a conservative modification to a SMILES string may cause a large effect in their 3D structure. Examples include the removal of brackets denoting substitution, such that a Y-shaped compound becomes linear, or the removal or changing of ring-closing locants. Therefore, optimization of molecular structures cannot be smooth in the space of 3D properties even though the SMILES strings change by only small amounts from iteration to iteration of the AI algorithm. We have previously presented a fragment-based generative approach (DeepFMPO) that addressed these modifications to the structure issue, albeit as 2D descriptions. Here, we introduce a significant extension to DeepFMPO, using detailed descriptions of 3D-properties to represent molecules more accurately.