In this work, we present the Porous Material Transformer (PMTransformer) model that combines multi-modal features from MOFs, COFs, PPNs, and zeolites. By pre-training on 1.9 million hypothetical porous materials, our model achieved state-of-the-art performance in predicting various properties of porous materials via fine-tuning. Furthermore, we introduced cross-material few-shot learning to address the challenge of asymmetry data aggregation in porous materials and proposed a method for predicting previously unexplored properties across different material classes (e.g. using MOF data to predict COF properties). Our approach provides an opportunity for understanding the underlying relationships between various classes of porous materials, allowing for the prediction of previously unexplored properties across these material classes, and thus facilitating a more comprehensive understanding and design of porous materials.