Although certain deep learning algorithms can perform well when trained on "smaller" datasets of tens of thousands of datapoints , these are more typically classifiers rather than generators. When we trained our models on the 35,806 datapoints from the StarPep database alone, we found that the models demonstrated poor reconstruction accuracy, low robustness, and high error. One method for solving this is to pretrain on a large corpus and fine-tune on a smaller one; however, we chose instead to train all at once on a larger corpus, as has previously been done for natural language models and chemical language models Gómez-Bombarelli et al. Since preliminary analysis on the StarPep database sequences indicated that 35,806 was not a large enough dataset to properly train our generative VAE-based models, we expanded the dataset by adding negative examples from Uniprot. After executing the query ) AND (keyword:(KW-0929))], we found 714 of unreviewed peptide sequences with the label "antimicrobial" and, placing "NOT" before the keyword, found 3,713,736 of unreviewed peptide sequences without the label "antimicrobial." We also found 1,108 of reviewed peptide sequences with the label "antimicrobial" and 11,941 reviewed peptides without the label "antimicrobial". Since we would not have been able to significantly expand our dataset by using solely reviewed sequences, we choose to include some non-reviewed sequences. We recognize that in the worse case it is possible that this may include a potentially substantial number of AMPs that have not been identified as such. We considered the possibility of utilizing an existing sequence predictor for AMPs to identify these possibly unlabeled AMPs; however, we decided against it to avoid introducing unknown assumptions into the data at this stage. Instead we consider a different goal for ordering the space. We employ a property predictor that classifies as hits peptides with experimentally-verified antimicrobial properties (which we label with integer 1) and as misses peptides without experimentally-verified antimicrobial properties (which we label with integer 0) and demonstrate that training such a predictor enforces an ordering of the space that allows generation of AMP-like sequences.