The Virtual Solar-Terrestrial Observatory (VSTO) Portal at vsto.org provides a set of guided workflows to implement use cases designed for solar-terrestrial physics and upper atmospheric science. Semantics are used in VSTO to model abstract instrument and parameter classifications, providing data access to users without extended domain specific vocabularies. The temporal restrictions used in the workflows are currently possible via RESTful services made to a remote system with access to a SQL-based metadata catalog. In order to provide a greater range of temporal reasoning and search capabilities for the user, we propose an alternative architecture design for the VSTO Portal, where the temporal metadata is integrated in the domain ontology. We achieve this integration by converting temporal metadata from the headers of raw data files into RDF using the OWL-Time vocabulary. This presentation covers our work with semantic temporal metadata, including: our representation using OWL-Time, issues that we have faced in persistent storage, and performance and scalability of semantic query. We conclude with discussions of the significance semantic temporal metadata has in virtual observatories.