I was afraid of forgetting him, or perhaps I did not know how I was going to continue to "spend time" with him.
J'avais tellement peur de l'oublier ou plutôt je ne savais pas comment j'allais continuer à le "fréquenter".