\section{Surrogate Model} \label{app:surrogate}
This section covers specific implementation details about the surrogate model as well as observations on its performance.
\subsection{Implementation Details}
\paragraph{Dataset} To create training data for the surrogate model $\hat{f}_\phi$, we used the phosphene model described in \cite{granley_computational_2021} and implemented in pulse2percept v0.8 \cite{beyeler_pulse2percept_2017}. $50,000$ stimuli were created by first selecting a number of electrodes to stimulate between 1 and 30 randomly chosen electrodes, then randomly selecting an amplitude between 1 and 10 (specified as a multiple of the assumed threshold current) and frequency between 1 and 200 Hz for each electrode. In addition, between 10 and 100 electrodes were chosen to act as ``noise'' electrodes, where either amplitude or frequency was given a nonzero value, but not both.
The purpose of these electrodes was for the surrogate model to learn that both a nonzero amplitude and a nonzero frequency are required to produce a visible percept.
We used an 80-20 train-test split.
As the surrogate model is highly dependent on patient-specific parameters $\phi$, we generated new data and fit a separate surrogate for each of the following $\phi$: ($(\rho, \lambda)\in\{(150,100), (150,1500), (800,100), (800,1500)\}$).

\paragraph{Network Architecture}
The surrogate model $\hat{f}_\phi$ used a fully-connected architecture. The input to the model was a stimulus matrix $\mathbf{s} \in \mathbb{R}_{\geq0}^{n_e \mathrm{x} 3}$, which was identical to the input to $f$.
The stimulus matrix was split into amplitude and frequency components (pulse duration was not used due to poor model performance), which were fed through a \ac{FC} layer.
The outputs of both \ac{FC} layers were concatenated and fed through another \ac{FC} layer. Concurrently, the model computed the element-wise product of the amplitude and frequency components and passed it through a separate \ac{FC} layer. The outputs of the previous two layers were then concatenated and fed through a final \ac{FC} layer with output size $49\times49$.

The model was trained for 45 epochs using AdamW \citep{loshchilov2017decoupled} optimizer and MAE loss.

\subsection{Approximating the Forward Model}

The surrogate model was able to accurately approximate the true phosphene model $f$. Table \ref{tab:surrogate} shows MAE over the validation set ($10,000$ percepts) for all 4 trained $\hat{f}_\phi$. Visually, the predicted percepts were nearly identical to the ground truth.

\begin{table}[h]
    \centering
    \caption{Surrogate model performance}
    \label{tab:surrogate}
    \resizebox{\columnwidth}{!}{%
    \begin{tabular}{@{}c|cccc@{}}
    \toprule
    $\phi$ & $\rho=150$ $\lambda=100$ & $\rho=150$ $\lambda=1500$ & $\rho=800$ $\lambda=100$ & $\rho=800$ $\lambda=1500$ \\ \midrule
    MAE    & 0.0119                  & 0.0189                   & 0.0078                  & 0.0115                   \\ \bottomrule
    \end{tabular}
    }
\end{table}

\subsection{Predicted Stimuli}

Despite the low surrogate validation error, training with the surrogate model would often result in the encoder suggesting almost adversarial stimuli; that is, stimuli that if fed through the true forward model $f$ would lead to drastically different percepts than if fed through the surrogate model $\hat{f}$ (see \ref{fig:true_vs_surrogate}). With these adversarial-like stimuli, the encoder appears to be performing well under the surrogate model, but performs poorly when the same stimuli are input to the true forward model. We identify this as the primary disadvantage of using a surrogate model and resolving this issue remains an open research problem for end-to-end training with surrogate methods.

We noticed several issues caused by the effects of varying stimulus parameters on phosphene appearance. For example, increasing amplitude increases size and brightness, while increasing frequency increases brightness only. We noticed a larger mismatch between the surrogate and the forward model on the extreme ends of the spectrum (\emph{e.g.} very high frequency, low amplitude), resulting in the encoder settling into a minimum that does not exist in the true forward model. It is important to note this disparity appears despite a high training accuracy of the surrogate alone.
Although these examples are specific to the bionic vision application, we expect surrogate models derived to describe other neuromodulation technologies to suffer from similar limitations.

\begin{figure}[h]
    \centering
    \includegraphics[width=\linewidth]{figures/appA_true_vs_surrogate.png}
    \caption{The encoder would often suggest stimuli that lead to drastically different percepts when fed through the surrogate model ($\hat{f}$, middle row) as compared to the true forward model ($f$, bottom row). Examples are shown for $\rho=800$; $\lambda=100$ (\emph{left}) and $\rho=800$ $\lambda=1500$ (\emph{right}).}
    \label{fig:true_vs_surrogate}
\end{figure}

\newpage
\section{Predicted Stimuli} \label{app:stimuli}
Here, we directly examine the stimuli resulting from \ac{HNA} and naive encoders. Stimuli and their resulting phosphenes for example images from the test set are shown in Figure \ref{fig:stimuli}. The naive encoder produces stimuli with constant frequency (20 Hz) and pulse duration (0.45 ms), which are not shown.

We make the following observations about the predicted stimuli:
\begin{itemize}[topsep=0pt, noitemsep, leftmargin=20pt]
    \item Both encoders activate electrodes corresponding to the shape of the target image. In naive stimuli, the amplitude directly corresponds to the pixel brightness. In \ac{HNA} stimuli, the distributions of amplitude, frequency, and pulse duration across the electrodes is more complex and harder to characterize, but lead to higher-quality phosphenes.
    \item \ac{HNA} uses amplitudes inversely proportional to $\rho$.
    \item For small $\rho$, \ac{HNA} primarily uses amplitude to control brightness. For large $\rho$, \ac{HNA} primarily uses frequency to modulate brightness, keeping amplitudes low to limit phosphene size.
    \item \ac{HNA} uses small pulse durations to create lines parallel to the underlying axon \ac{NFB} (\textit{i.e.}, it utilizes the streaked phosphenes to its advantage), and large pulse durations to create lines perpendicular to the underlying \ac{NFB}. In other words, \ac{HNA} was able to exploit application-specific (\emph{i.e.}, neuroanatomical) information that is baked into the forward model.
    \item On average, \ac{HNA} uses more electrodes, larger frequencies and pulse durations, and smaller amplitudes than the naive encoder. A large active electrode count and high pulse durations may not be desirable for some prostheses, due to tissue activation and frame rate limits. We found that it was easy to constrain these parameters using regularization on the output stimuli, at the cost of slightly decreased performance.
\end{itemize}
\begin{figure}[h!]
    \centering
    \includegraphics[width=0.9\linewidth]{figures/appB_combined.png}
    \caption{\emph{Top}: Example MNIST target images, and the phosphenes produced by \ac{HNA} and naive encoders, encoded at various $\rho$ and $\lambda$ values. \emph{Center}: The stimuli corresponding to the \ac{HNA} phosphenes. From top to bottom, stimulus frequency (Hz), amplitude (xTh), and pulse duration (ms) are shown. The number of 'active' electrodes stimulated above threshold levels is given below each stimuli. \emph{Bottom}: Stimuli corresponding to the naive phosphenes.}
    \label{fig:stimuli}
\end{figure}

\newpage

\section{COCO Dataset} \label{app:coco_dataset}
For the COCO task (Section \ref{sec:coco}), we used subset of images from the MS-COCO dataset \cite{lin_microsoft_2015}. MS-COCO was chosen due to its selection of common household objects relevant to the daily life of prosthesis users, as well as availability of ground-truth segmentation masks. To select the images suitable for prosthetic vision, we filtered out images according to the following criteria:
\begin{itemize}[topsep=0pt, noitemsep, leftmargin=20pt]
    \item[1.] \textbf{Too cluttered.} Any image with greater than 15 total objects was removed. Removed: 15566
    \item[2.] \textbf{Select chosen objects.} Any image that did not have at least 1 object from the selected categories that was larger than 4\% of the total image was removed. Removed: 42289
    \item[3.] \textbf{Too many.} Any image with greater than 5 objects meeting criteria 2 was removed. Removed: 1017
    \item[4.] \textbf{Too dim.} Any objects in the image with average pixel brightness less than 50 were discarded. If this resulted in an image having 0 remaining objects, the image was removed. Removed: 434
\end{itemize}
\begin{figure}[ht!]
    \centering
    \includegraphics[width=\linewidth]{figures/fig-appB_cococlasses.png}
    \caption{Number of images (\emph{left}) or instances (\emph{right}) of each category in the processed COCO dataset.}
    \label{fig:coco_classes}
\end{figure}

This resulted in a total of 47532 training images and 11883 test images (80-20 train-test split). The objects in the remaining images were segmented out using the ground-truth segmentation masks, resized to (49, 49), and converted to grayscale. The distributions of classes used is shown in Figure \ref{fig:coco_classes}.

\newpage

\section{Hyperparameter Selection}\label{app:hyperparameter}
In this section, we detail how \ac{HNA} hyperparameters ($l$, $k$, $\alpha$, and $\beta$) were chosen.

\paragraph{VGG Loss}
To choose the layer of the VGG network to use for VGG loss ($l$, Eq \ref{eq:vgg}) we performed cross validation across a set of candidate layers. Previous studies \cite{li_universal_2017} have shown that the first layer with each of the 5 convolutional blocks perform well for neural style transfer. Thus, we choose these as our candidate layers. For cross validation, we trained \ac{HNA} for 50 epochs using each candidate layer. The resulting phosphenes are shown in Figure \ref{fig:crossval}. Using earlier layers, the VGG term performs similarly to \ac{MAE}, and phosphenes are disconnected. We chose layer 5\_1 based on its perceived ability to capture high-level perceptual differences between images, although layer 4\_1 also performs similarly.

\begin{figure}[ht!]
    \centering
    \includegraphics[width=\linewidth]{figures/appD_crossval.png}
    \caption{Phosphenes produced by \ac{HNA} encoder with different layers chosen for VGG loss. Layer 5\_1 denotes the first layer within the fifth convolutional block.}
    \label{fig:crossval}
\end{figure}

\paragraph{Laplacian Smoothing}
We chose to use a kernel size 5 for the Laplacian filter used to estimate the second derivative ($k$, Eq. \ref{eq:smooth}). The size of the filter controls the scale on which smoothing is applied (\textit{i.e.}, smaller filters sizes only encourage continuity within a small local region, whereas larger filters encourage continuity within a larger region). Size 5 was chosen because larger filters were observed to over-smooth the image, while smaller filters still led to highly disconnected phosphenes.

\paragraph{Joint Perceptual Metric}
We performed cross validation to find the best values for $\alpha$ and $\beta$. Instead of using one value, we found scheduled weighting to be crucial for performance. The scheduler incrementally increased the weight of the VGG loss ($\beta$) from 0 while simultaneously decreasing the initially high weight on the smoothing constraint ($\alpha$).
This was motivated by the observation that the VGG loss performed poorly during early iterations when the predicted phosphene was near-random.

Under this scheduled weighting strategy, the loss is dominated early on by the \ac{MAE} and smoothing terms. This encourages the the model to just output reasonable encodings. As training progresses, the predicted phosphenes become higher quality, causing the VGG loss to perform better, and thus the smoothing term is no longer as important.

Additionally, we found it beneficial to temporarily decrease the learning rate by a factor of 10 for a short 'warm-up' duration following each increase in $\beta$, before resetting to 50\% of the prior learning rate. This results in the learning rate gradually decreasing throughout training by a factor of around 100. Throughout the paper, we use $\alpha=0$ and $\beta=0.00008$ for comparisons of loss values.

\newpage
\section{Modeling Other Patient-to-Patient Variations} \label{app:other_modelparams}
Previously, results were presented across patient-specific parameters $\rho$ and $\lambda$, because these have the greatest impact on phosphene appearance. However, the forward model has a number of other patient-specific parameters, which \ac{HNA} is also able to adapt to. For full details on all parameters of the forward model, see \cite{granley_computational_2021}. Out of the remaining parameters, $a2$, $a3$, and $a5$ are the most impactful on phosphene appearance. $a2$ and $a3$ modulate how much the brightness contribution from each electrode scales with increasing amplitude and frequency, respectively. $a5$ locally scales the global radial current spread $\rho$ based on each electrodes amplitude. Figure \ref{fig:modelparams} (\emph{left}) illustrates the effect of these parameters on phosphene appearance.

Figure \ref{fig:modelparams} compares \ac{HNA} to naive encoder performance across $a2$, $a3$, and $a5$. The ranges for these parameters are based on values empirically observed in retinal prosthesis users \cite{granley_computational_2021}. \ac{HNA} produces relatively consistent phosphenes, and outperforms the naive encoder across all conditions.

\begin{figure}[ht!]
    \centering
    \includegraphics[width=\linewidth]{figures/appE_modelparams.png}
    \caption{\emph{Left}: Examples of how a2, a3, and a5 affect single-electrode phosphenes. a2 modulates local brightness scaling with increasing amplitude, a3 modulates local brightness scaling with increasing frequency, and a5 modulates local size scaling with increasing amplitude. \emph{Center}: Phosphenes predicted with \ac{HNA} and naive encoders for varying a2, a3, and a5, increasing left to right. \emph{Right}: Plot showing the joint loss across a2, a3, and a5 for \ac{HNA} (solid) and naive encoder (dashed line).}
    \label{fig:modelparams}
\end{figure}

\newpage

\section{COCO Patient-to-Patient Variations} \label{app:coco_patient}
We repeated the analysis presented in Section \ref{sec:patient-to-patient} for the COCO dataset. Figure \ref{fig:coco_patient-to-patient}A shows two example COCO images, encoded by both \ac{HNA} and the naive encoder, across varying $\rho$ and $\lambda$ values. The heatmaps in Figure \ref{fig:coco_patient-to-patient}B show the log of the joint perceptual loss across simulated patients, for both the naive and \ac{HNA} encoders. To measure phosphene consistency, we performed T-SNE clustering on a subset of the COCO images which have only 1 object. Unfortunately, T-SNE clustering of the ground-truth COCO images did not form groups corresponding to the object types (Figure \ref{fig:coco_patient-to-patient}C), suggesting that the representation of object instances vary drastically across COCO images. Therefore, it was not meaningful to repeat the analysis presented in Fig.~\ref{fig:mnist-rho-lam}C.

Similar to the MNIST results presented in Section \ref{sec:patient-to-patient}, \ac{HNA} produced higher-quality representations than the naive encoder, resulting in a lower joint loss for every simulated patient. \ac{HNA} performed consistently well across all simulated patients (Figure \ref{fig:coco_patient-to-patient}B), with a small increase in loss for small $\rho$ ($< 100$). Similar to MNIST, the naive encoder only performs well for patients with a mid-to-low $\rho$ ($\approx 200$) and low $\lambda$.

\begin{figure}[ht!]
    \centering
    \includegraphics[width=\linewidth]{figures/appF_coco_combined.png}
    \caption{COCO Encoder performance across simulated patients (varying $\rho$ and $\lambda$). \textbf{A:} Phosphenes produced by HNA and Naive encoders of two example images. \textbf{B:} Heatmaps showing the log joint loss across $\rho$ and $\lambda$ for \ac{HNA} and naive encoders. \textbf{C.} Ground-truth COCO images cannot be clustered using T-SNE into groups corresponding to the object types. The clustering was performed on COCO images that only contained one object.}
    \label{fig:coco_patient-to-patient}
\end{figure}

\newpage

\section{Simulating Higher-Resolution Implants} \label{app:high-res}
On the COCO task, \ac{HNA} significantly outperformed the naive encoder, but was still unable to capture all of the detail in the images. Two of the main reasons for this are the limited spatial resolution of the implant and the patient-specific distortions from the forward model. Here, we present results from \ac{HNA}s trained on implants of higher resolution, at small $\rho$ and $\lambda$. The chosen implants are illustrated in Figure \ref{fig:implants}A. For a fair comparison, each \ac{HNA} was trained for only 50 epochs.

Phosphenes resulting from the \ac{HNA} trained on the different implants are shown in Figure \ref{fig:implants}C, and the losses across implants is plotted in Figure \ref{fig:implants}B. As implant resolution increases, the phosphenes look increasingly similar to the ground truth, and small details (e.g. facial details, textures) start to emerge.

Thus, \ac{HNA}s initial failure to capture high-frequency details in the image appears to be an application-specific limitation for visual prostheses more so than a limitation of the \ac{HNA} framework. For visual prostheses, learning to reconstruct the high-frequency features of complex images despite distortions and limited implant resolution remains an open problem.

\begin{figure}[h]
    \centering
    \includegraphics[width=\linewidth]{figures/appG_combined.png}
    \caption{\textbf{A}: The 4 different implants compared. The main text uses the $15 \times 15$ implant. \textbf{B}: The joint perceptual loss of \ac{HNA}s trained on the different implants after 50 epochs. \textbf{C}: Example images showing the reconstructed phosphenes using each implant}
    \label{fig:implants}
\end{figure}
\section{Introduction}

Sensory neuroprostheses are emerging as a promising technology to restore lost sensory function or augment human capacities \cite{cinel_neurotechnologies_2019,fernandez_development_2018}.
In such devices, diminished sensory modalities (e.g., hearing \cite{wilson_better_1991}, vision \cite{fernandez_visual_2021,luo_argus_2016}, cutaneous touch \cite{tan_neural_2014}) are re-enacted through streams of artificial input to the nervous system.
For example, visual neuroprostheses electrically stimulate neurons in the early visual system to elicit neuronal responses that the brain interprets as visual percepts.
Such devices have the potential to restore a rudimentary form of vision to millions of people living with incurable blindness.

However, all of these technologies face the challenge of interfacing with a highly nonlinear system of biological neurons whose role in perception is not fully understood. Due to the limited resolution of electrical stimulation, prostheses often create neural response patterns foreign to the brain.
Consequently, sensations elicited by current sensory neuroprostheses often appear artificial and distorted \cite{erickson-davis_what_2021,murray_embodiment_2008}.
A major outstanding challenge is thus to identify a stimulus encoding that leads to perceptually intelligible sensations.
Often there exists a numeric or symbolic forward model, $f$ (Fig.~\ref{fig:overview}A), constrained by empirical data, that can predict a neuronal or (ideally) perceptual response to the applied stimulus (see \cite{brunton_data-driven_2019} for a recent review).
To find the stimulus that will elicit a desired response, one essentially needs to find the inverse of the forward model, $f^{-1}$. However, realistic forward models are rarely analytically invertible, making this a challenging open problem for neuroprostheses.

Here we propose to approach this as an end-to-end optimization problem, where a deep neural network (\emph{encoder}) is trained to invert a known, fixed forward model (\emph{decoder}, Fig.~\ref{fig:overview}B).
The encoder is trained to predict the patterns of electrical stimulation patterns that elicit perception (\emph{e.g.,} vision, audition) or neural responses (\emph{e.g.,} firing rates) closest to the target. This \ac{HNA} could in theory be used to optimize stimuli for any open-loop neuroprosthesis with a known forward model that approximates the underlying biological system.

In order to optimize end-to-end, the forward model must be differentiable and computationally efficient.
When this is not the case, an alternative approach is to train a surrogate neural network, $\hat{f}$, to approximate the forward model \cite{relic_deep_2022, montes_de_oca_zapiain_accelerating_2021, nabian_deep_2019, nikolopoulos_non-intrusive_2022}. However, even well-trained surrogates may result in errors when included in our end-to-end framework, due to the encoders' ability to learn to exploit the surrogate model. We therefore also evaluate whether a surrogate approach to \ac{HNA} is suitable.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/fig-overview.png}
    \caption{\emph{A)} Sensory neuroprosthesis. A numeric or symbolic forward model ($f$) is used to approximate the neuronal or, ideally, perceptual response to electrical stimuli.
    \emph{B)} \Acf{HNA}. A deep neural encoder ($f^{-1}$) is trained to predict the patterns of electrical stimulation that elicit responses closest to the target.
    \emph{C)} Bionic vision. Visual neuroprostheses are one prominent application of \acs{HNA}, where electrical stimulation is used to elicit a desired visual percept.}
    \label{fig:overview}
\end{figure}

To this end, we make the following contributions:
\begin{itemize}[topsep=0pt,leftmargin=15pt]
    \item We propose a \acf{HNA} consisting of a deep neural encoder trained to invert a fixed, numerical or symbolic forward model (decoder) to optimize stimulus selection. Our framework is general and addresses an important challenge with sensory neuroprostheses.

    \item As a proof of concept, we demonstrate the utility of \ac{HNA} for visual neuroprostheses, where we predict electrode activation patterns required to produce a desired visual percept.
    We show that the \acs{HNA} is able to produce high-fidelity, patient-specific stimuli representing handwritten digits and segmented images of everyday objects, drastically outperforming conventional approaches.

    \item We evaluate replacing a computationally expensive or nondifferentiable forward model with a surrogate,

    highlighting benefits and potential dangers of popular surrogate techniques.
\end{itemize}

\section{Background}

\paragraph{Sensory Neuroprostheses}
Recent advances in neural understanding, wearable electronics, and biocompatible materials have accelerated the development of sensory neuroprostheses to restore perceptual function to people with impaired sensation.
Use of neuroprostheses typically requires invasive implants that elicit neural responses via electrical, magnetic, or optogenetic stimulation. Two of the most promising applications include cochlear implants, which stimulate the auditory nerve to elicit sounds \cite{wilson_better_1991}, and visual implants (see next subsection) to restore vision to the blind. However, a variety of other devices are in development; for instance, to restore touch \citep{tan_neural_2014, tabot_restoring_2013} or motor function \citep{capogrosso_brainspine_2016}.
The latter differ from other sensory neuroprostheses in that they generate stimuli using motor feedback (\emph{closed-loop}) \citep{chapman_multifunctional_2018, wagner_targeted_2018}.
In the absence of feedback (\emph{open-loop}), a proper stimulus encoding is paramount to the success of these devices.

\paragraph{Restoring Vision to the Blind}
For millions of people who are living with incurable blindness, a visual prostheses (\emph{bionic eye}, Fig.~\ref{fig:axonmapmodel}, \emph{left}) may be the only treatment option \cite{fernandez_visual_2021}.
Analogous to cochlear implants, these devices electrically stimulate surviving cells in the visual pathway to evoke visual percepts (\emph{phosphenes}), which can support simple behavioral tasks
\cite{luo_argus_2016,stingl_interim_2017,karapanos_functional_2021}.

A common misconception is that each electrode in the microelectrode array can be thought of as a pixel in an image; to generate a complex visual experience, one then simply needs to turn on the right combination of pixels \cite{dobelle_artificial_2000}.
However, recent evidence suggests that phosphenes often appear distorted (\emph{e.g.}, as simple geometric shapes such as lines, wedges, and blobs) and vary drastically across subjects and electrodes \cite{erickson-davis_what_2021,fernandez_visual_2021}.

Phosphene appearance has been best characterized in epiretinal implants, where inadvertent activation of \acp{NFB} in the optic fiber layer of the retina leads to elongated phosphenes \cite{beyeler_model_2019,rizzo_perceptual_2003} (Fig.~\ref{fig:axonmapmodel}, \emph{center}).
To this end, Granley \emph{et.~al} \citep{granley_computational_2021} developed a forward model to predict phosphene shape as a function of both neuroanatomical parameters (\emph{i.e.}, retinal location of the stimulating electrode) and stimulus parameters (\emph{i.e.}, pulse frequency, amplitude, and duration).
With this model, phosphenes are primarily characterized by two main parameters, $\rho$ and $\lambda$, which dictate the size and elongation of the elicited phosphene, respectively (Fig.~\ref{fig:axonmapmodel}, \emph{right}).
Stimulation from multiple electrodes is nonlinearly integrated into a combined perception, and if two electrodes happen to activate the same \ac{NFB}, they might not generate two distinct phosphenes.
In addition, $\rho$ and $\lambda$ vary drastically across patients \cite{beyeler_model_2019}, making stimulus encoding a particularly challenging problem.

\begin{figure}[t]
    \centering
    \includegraphics[width=.9\linewidth]{figures/fig-axon-map.png}
    \caption{
        \emph{Left}: Visual prosthesis. Incoming target images are transmitted from a camera to an implant in the retina, which encodes the image as an electrical stimulus pattern. \emph{Center}: Electrical stimulation (red disc) of a \acf{NFB} (gray lines) leads to elongated tissue activation (gray shaded region). The resulting phosphene appears elongated as well; its shape can be described by two parameters, $\lambda$ (axonal spread) and $\rho$ (radial spread).
        \emph{Right}: Predicted visual percepts for a particular MNIST digit for varying $\rho$ and $\lambda$ values.

        }
    \label{fig:axonmapmodel}
\end{figure}

\section{Related Work}

The conventional `naive' encoding strategy simply sets the amplitude of each electrode to the brightness of the corresponding pixel in the target image \cite{chen_simulating_2009,luo_argus_2016,spencer_global_2019}, essentially making the stimulus a downsampled version of the target.
Although simple, this encoding strategy only works with near-linear forward models and cannot account for phosphene distortions encountered by real prosthesis patients \cite{beyeler_model_2019,erickson-davis_what_2021}.
When used with a more sophisticated forward model, this strategy often produces unrecognizable percepts (Fig.~\ref{fig:axonmapmodel}, \emph{right}).

To provide an alternative,  Shah \textit{et al.}~\cite{shah_optimization_2019} used a greedy approach to iteratively select the stimuli that best recreate a desired neural activity pattern over a given temporal window, assuming that the brain would integrate them into a coherent visual percept. Ghaffari \textit{et al.}~\citep{ghaffari_closed-loop_2021} used a neural network surrogate model combined with an interior point algorithm to optimize for localized, circular neural activation patterns for individual electrodes. Fauvel \textit{et al.}~\cite{fauvel_human---loop_2021} used human in-the-loop Bayesian optimization to achieve encodings perceptually favored by the subject.
Spencer \textit{et al.}~\cite{spencer_global_2019} proposed framing the problem of stimulus encoding as inversion of a forward model of neural activation patterns. To approximate the inverse, their approach either requires simplification or is NP-hard \cite{spencer_global_2019}.
Furthermore, it cannot predict the perceptual consequences of the predicted neural activity.

van Steveninck \textit{et al.} \cite{de_ruyter_van_steveninck_end--end_2022} proposed an end-to-end optimization strategy with a fixed phosphene model, similar to \ac{HNA}. However, their approach crucially differs from ours in that they included a secondary deep neural network to post-process the predicted phosphenes. This is a critical limitation, because a low reconstruction loss does not reveal whether a high-fidelity encoder was learned or whether the secondary decoder network simply learned to correct for the encoder's mistakes.
In addition, they used an unrealistic phosphene model that simply upscales and smooths a binary stimulus pattern.
It is therefore not clear whether their results would generalize to real visual prosthesis patients.

Relic \textit{et al.}~\cite{relic_deep_2022} also utilized the end-to-end approach, but without the secondary decoder network used in \cite{de_ruyter_van_steveninck_end--end_2022}. They used a more realistic phosphene model, which accounts for some spatial distortions (\textit{e.g.}, axonal streaks), but not the effects of stimulus parameters. Since including a nontrivial phosphene model in the loop is not straightforward, they instead trained a surrogate neural network to approximate the forward model. We re-implemented Relic's `surrogate' approach in this paper as a baseline method to compare against, as described in Section~\ref{sec:forward}.

Taken together, we identified three main limitations of previous work that this study aims to address:
\begin{itemize}[topsep=0pt,leftmargin=15pt]
    \item[1)] \textbf{Unrealistic forward models.} Most previous approaches (\emph{e.g.}, \cite{de_ruyter_van_steveninck_end--end_2022,shah_optimization_2019,spencer_global_2019}) use an overly simplified forward model that cannot account for empirical data \cite{beyeler_model_2019,erickson-davis_what_2021}.
    We overcome this limitation by developing (and inverting) a differentiable version of a neurophysiologically informed and psychophysically validated phosphene model \cite{granley_computational_2021} that can account for the effects of stimulus amplitude, frequency, and pulse duration on phosphene appearance.
    \item[2)] \textbf{Optimization of neural responses.}
    Most previous approaches (\emph{e.g.}, \cite{shah_optimization_2019,spencer_global_2019}) focus on optimizing neural activation patterns in the retina in response to electrical stimulation (``bottom-up'').
    However, the visual system undergoes extensive remodeling during  blinding diseases such as retinitis pigmentosa \cite{marc_remodeling_2003}.
    Thus the link between neural activity and visual perception is unclear.
    We overcome this limitation by inverting a phenomenological (``top-down'') model constrained by behavioral data that predicts visual perception directly from electrical stimuli \cite{beyeler_model_2019,granley_computational_2021}.

    \item[3)] \textbf{Objective function}
    Most previous approaches rely on minimizing \ac{MSE} between the target and reconstructed image.
    Although simple and efficient, \ac{MSE} is known to be a poor measure of perceptual dissimilarity for images \citep{wang_mean_2009} and does not align well with human assessments of image quality \citep{zhai_perceptual_2020}.
    We overcome this limitation by proposing a joint perceptual metric that combines \ac{MAE}, VGG, and Laplacian smoothing losses.
\end{itemize}

\section{Methods}

\paragraph{Problem Formulation}
We consider a system where there is some known forward process $f$ mapping stimuli to responses $f : \mathcal{S} \mapsto \mathcal{R}, f(\mathcal{S}) \subset \mathcal{R}$.
In the case of visual prostheses, $f$ may map stimuli to visual percepts. In spinal cord rehabilitation devices, $f$ might map electrical stimuli to neural activation patterns. $f$ may be parameterized by patient-specific parameters $\phi$.

Finding the best stimulus for an arbitrary target response $\mathbf{t} \in \mathcal{R}$ is therefore equivalent to finding the inverse of $f$.
However, since not every response can be perfectly reproduced by a stimulus, the true inverse of $f$ is not well defined.
We therefore seek the pseudoinverse (still denoted as $f^{-1}$ for simplicity) instead, which outputs the stimuli that produce the closest response to $\mathbf{t}$ under some distance metric $d$:
\begin{equation}
    f^{-1}(\mathbf{t}, \phi) \coloneqq \underset{\mathbf{s}\in\mathcal{S}}{\argmin}\,d(f(\mathbf{s}; \phi), \mathbf{t}).
\end{equation}

This problem is straightforward to solve using an autoencoder approach, where a learned encoder $f^{-1}$ is trained to invert the fixed decoder $f$ (i.e., forward model).

\paragraph{Encoder}
We approximate the pseudoinverse $f^{-1}$ with a deep neural network encoder $\hat{f}^{-1}(\mathbf{t}, \phi; \theta)$ with weights $\theta$, trained to minimize the distance $d$ between the target image $\mathbf{t}$ and predicted image $\hat{\mathbf{t}}$:
\begin{equation}
   \underset{\theta,\,\phi \sim p(\phi)}{\min}\,\,  d(\mathbf{t},\hat{\mathbf{t}})
\end{equation}
\begin{equation}
    \hat{\mathbf{t}}  = f(\,\hat{f}^{-1}(\mathbf{t}, \phi;\, \theta);\,\, \phi),
\end{equation}
where $\phi$ is sampled from a uniform random distribution spanning the empirically observed range of patient-specific parameters \cite{beyeler_model_2019, granley_computational_2021}.

We use a simple architecture consisting solely of \ac{FC} and \ac{BN} \cite{ioffe_batch_2015} layers. First, the target $\mathbf{t}$ is flattened and input to one \ac{FC} layer. In parallel, the patient parameters $\phi$ are input to a \ac{BN} layer before two hidden \ac{FC} layers. Then, the outputs of these two paths are concatenated, and the combined vector input through two \ac{FC} layers, producing an intermediate representation $\mathbf{x}$. Amplitudes are predicted from $\mathbf{x}$ with a \ac{FC} layer. The amplitudes are then concatenated with $\mathbf{x}$, put through a \ac{BN} layer, and used to predict frequency and pulse duration, each with a \ac{FC} layer.
The encoder outputs are merged into one stimulus matrix $\mathbf{\hat{s}}$.
All layers uses a leaky ReLU activation functions, except for stimulus outputs, which use ReLu activation, enforcing nonnegativity. In total there were 1.4M trainable parameters. We also tried using convolutional, transformers, and other attention layers, but the fully connected architecture performed best.

\paragraph{Decoder} \label{sec:forward}
For our decoder $f$, we use a reformulated but equivalent version of the model described in \cite{granley_computational_2021}. We use a square $15 \times 15$ array of 150$\mu$m electrodes, spaced 400$\mu$m apart and centered on the fovea. The size and scale of this device was motivated by similar designs in real epiretinal implants.

$f$ takes as input a stimulus matrix $\mathbf{s} \in \mathbb{R}_{\geq0}^{n_e \mathrm{x} 3}$, where the stimulus on each electrode ($\mathbf{s_e}$) is a biphasic pulse train described by its frequency, amplitude, and pulse duration. A simulated map of retinal \ac{NFB}s is used to predict phosphene shape. Following \cite{beyeler_model_2019}, each retinal ganglion cells' activation is assumed to be the maximum stimulation intensity along its axon. Each axon's sensitivity to stimulation is assumed to decay exponentially with i) Euclidean distance to the stimulating electrode (radial decay rate, $\rho$) and distance to the soma along the curved axon (axonal decay rate, $\lambda$).
To account for stimulus properties \citep{granley_computational_2021}, $\rho$, $\lambda$, and the per-electrode brightness are scaled by three simple equations $F_\mathrm{size}(\mathbf{s_e}, \phi)$, $F_\mathrm{streak}(\mathbf{s_e}, \phi)$, and $F_\mathrm{bright}(\mathbf{s_e}, \phi)$, respectively.

The brightness of a pixel located at the point $\mathbf{x} \in \mathbb{R}^2$ in the output image is given by
\begin{equation} \label{eq:forward1}
    f(\mathbf{s}; \phi) = \max_{\mathbf{a} \in A}\sum_{e \in E}F_\mathrm{bright}(\mathbf{s_e}, \phi)  \exp\left(\frac{-|| \mathbf{x} - \mathbf{e}||_2^2}{2\rho^2 F_\mathrm{size}(\mathbf{s_e}, \phi) } + \frac{-d_{s}(\mathbf{x}, \mathbf{a})^2}{2\lambda^2 F_\mathrm{streak}(\mathbf{s_e}, \phi) }\right)
\end{equation}
where $A$ is the cells' axon trajectory, $E$ is the set of electrodes, $\phi = \{\rho, \lambda, ...\}$ is a set of 12 patient-specific parameters,
and $d_s$ is the path length along the axon trajectory \cite{jansonius_mathematical_2009}from $\mathbf{a}$ to $\mathbf{x}$:
\begin{equation} \label{eq:forward2}
    d_s(\mathbf{x}, \mathbf{a}) = \int_{\mathbf{a}}^\mathbf{x} \sqrt{A(\theta)^2 + \left(\frac{dA(\theta)}{d\theta}\right)^2}d\theta.
\end{equation}

This model ($f$) can be fit to individual patients; however, it is computationally slow and not differentiable.
For more details on these equations, see \citep{granley_computational_2021}.
We therefore considered two approaches.
\begin{itemize}[topsep=0pt, leftmargin=15pt]
    \item \textbf{Differentiable Model:} We reformulated equations \ref{eq:forward1} and \ref{eq:forward2} into an equivalent set of parallelized matrix operations, implemented in Tensorflow \cite{tensorflow2015-whitepaper}. Significant efforts were put towards developing a model optimized for XLA engines on GPU, resulting in speedups of up to 5000x compared to the model as presented in \cite{granley_computational_2021}, enabling large-scale gradient descent. To enforce differentiability, we numerically approximated $d_s$ using $|A|=500$ axon segments per axon.
    \item \textbf{Surrogate Model:} We also implemented the surrogate approach from \cite{relic_deep_2022} as a baseline method, where $f$ is approximated with another deep neural network $\hat{f}_\phi(\mathbf{s}; \theta_f)$ with weights $\theta_f$.
    To achieve this we generated 50,000 percepts using randomly selected stimuli passed through $f$ and fit a deep neural network to produce identical images. As $f$ is highly dependent on patient-specific parameters $\phi$, we generated new data and fit a separate surrogate model for each $\phi$ in our experimental set.
    Specific implementation details of the surrogate are presented in Appendix \ref{app:surrogate}. Our implementation improves upon \cite{relic_deep_2022} by using the more advanced phosphene model described above, which accounts for effects of stimulus properties and allows for optimization of stimulus frequency in addition to amplitude.
\end{itemize}

\paragraph{Metrics}
To measure perceptual similarity, we use a joint perceptual objective consisting of a VGG \citep{simonyan_very_2015-1} similarity term, a \acf{MAE} term, and a smoothness regularization term. The \ac{MAE} term is given by
\begin{equation} \label{eq:mae}
    L_{MAE} = ||\mathbf{t} - \hat{\mathbf{t}}||_1.
\end{equation}

The VGG term aims to capture higher-level, abstract differences between images \cite{de_ruyter_van_steveninck_end--end_2022, li_universal_2017}. Both the target image and the reconstructed phosphene are input to a VGG-19 network pretrained for classification on ImageNet \citep{deng_imagenet_2009}, and the MSE between the activations on a downstream convolutional layer is computed. Let $V_{l}$ be a function that extracts the activations of the $l$-th convolutional layer for a given image. The VGG loss is then defined as:
\begin{equation} \label{eq:vgg}
    L_{VGG} = ||V_{l}(\mathbf{t}) - V_{l}(\hat{\mathbf{t}})||_2^2.
\end{equation}

We also include a smoothing regularization term. This term imposes a loss on the second spatial derivative of the predicted image. A low second derivative implies that where the target image does change, it changes slowly. We found this encouraged smoother, more connected phosphenes.
To approximate the second derivative, we convolve the image with a Laplacian filter \citep{paris_local_nodate-1} of  size $k$, denoted by $Lap(\cdot, k)$, and compute the mean. The smoothness loss is given by:
\begin{equation} \label{eq:smooth}
    L_{Smooth} = \frac{1}{|\hat{\mathbf{t}}|} \sum_i(\frac{\partial^2}{dx^2} \hat{\mathbf{t}})_i
     =  \frac{1}{|\hat{\mathbf{t}}|} \sum_iLap(\hat{\mathbf{t}}, k)_i.
\end{equation}

Our final objective is the weighted sum of the three individual losses, given by Eq. \ref{eq:loss}, where $\alpha$ and $\beta$ are hyperparameters controlling the relative weighting of the three terms:
\begin{equation} \label{eq:loss}
    d = L_{MAE} + \alpha L_{Smooth} + \beta L_{VGG}.
\end{equation}

We also implement a secondary metric to quantify phosphene recognizability, applicable only for the MNIST reconstruction task. We first pre-train a classifier network on the MNIST targets until it reaches 99\% test accuracy, and then fix the weights. The \ac{RA} is then defined as the ratio of the classifiers accuracy on the reconstructed images to its accuracy on the targets $RA = ACC / ACC(\mathbf{t})$. A perfect encoder would result in $RA=100\%$. A similar process was not possible for the COCO task due to the possibility of having multiple objects in each target image.
\paragraph{Training/Optimization} \label{sec:training}
We trained using Tensorflow 2.7 \cite{tensorflow2015-whitepaper} on a single NVIDIA RTX 3090 with 24GB memory. Stochastic gradient descent with Nesterov momentum was used to minimize the joint perceptual loss. We used a batch size of 16 due to memory limitations imposed by $f$. The amplitude, frequency predictions are scaled by 2, 20 respectively, while the pulse duration predictions were shifted by 1e-3 prior to being fed through the decoder. This encourages the initial predictions of the network to be in a reasonable range. The Laplacian filter size $k$ is set to 5. We choose $l$ to be first convolutional layer in the last block using cross validation (see Appendix \ref{app:hyperparameter}).
Similarly, we perform cross validation to find the best values for $\alpha$ and $\beta$. Instead of using one value, we found that incrementally increasing the weighting of the VGG loss ($\beta$) from 0 while simultaneously decreasing the initially high weight on the smoothing constraint ($\alpha$) was crucial for performance, especially when the range of allowed $\phi$ values was large (see Appendix \ref{app:hyperparameter}).

\paragraph{Datasets}
We first evaluated on handwritten digits from MNIST \cite{deng2012mnist}, enabling comparison to previous works \cite{relic_deep_2022}. Images preprocessing consisted of resizing the target images to the same shape as the output of $f$ (49x49).
We also evaluate on more realistic images of common objects from the MS-COCO \citep{lin_microsoft_2015} dataset. We selected a subset of 25 of the MS-COCO object categories deemed more likely to be encountered by blind individuals (\textit{e.g.} people, household objects), and use only images that contain at least one instance of these objects. We further filter out images by various other criteria, such as being too cluttered or too dim. This process results in a total of approximately 47K training images and 12K test images. See Appendix \ref{app:coco_dataset} for a full description of the selection process.

Natural images often contain too much detail to be encoded with prosthetic vision. While scene simplification strategies exist \citep{han_deep_2021}, we focus on the encoding algorithm,
so we simply used the ground-truth segmentation masks to segment out the objects of interest. The images were then converted to grayscale, and resized to (49, 49).

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figures/fig-mnist-performance.png}
    \caption{Reconstructed MNIST targets for \ac{HNA}, surrogate, and naive encoders across 4 specific simulated patients. Note that the brightness of the naive encoder is clipped for display}
    \label{fig:mnist-performance}
\end{figure}

\section{Results}
\subsection{MNIST}
The phosphenes produced from the \ac{HNA}, surrogate, and naive encoders on the MNIST test set are shown in Fig.~\ref{fig:mnist-performance} and performance is summarized in Table~\ref{tab:mnist}.
For each MNIST sample, the target image is input to the encoder, which predicts a stimulus. The stimulus is fed through the true forward model $f$, and the predicted phosphene is shown. Since the surrogate method must be retrained for each $\phi$, results are only shown for 4 simulated patients.
Our proposed approach outperformed the baselines across all metrics
(see Appendix \ref{app:stimuli} for a comparison of stimuli).

\begin{table}[t]
    \centering
    \vspace{6pt}
    \caption{MNIST performance}
    \label{tab:mnist}
    \resizebox{\columnwidth}{!}{%
    \begin{tabular}{@{}ccccccccccccc@{}}
        \toprule
        Encoding  & \multicolumn{3}{c}{$\rho$=150  $\lambda$=100} & \multicolumn{3}{c}{$\rho$=150  $\lambda$=1500} & \multicolumn{3}{c}{$\rho$=800  $\lambda$=100} & \multicolumn{3}{c}{$\rho$=800  $\lambda$=1500} \\ \midrule
                  & Joint Loss      & MAE & RA   & Joint Loss   & MAE    & RA     & Joint Loss     & MAE   & RA     & Joint Loss    & MAE     & RA         \\ \cmidrule(l){2-4} \cmidrule(l){5-7} \cmidrule(l){8-10} \cmidrule(l){11-13}
        Naive     & 1.161           & 0.1855 & 90.3 & 1.442        & 0.214  & 78.1   & 8.152          & 1.500 & 34.8   & 8.780         & 1.726      & 28.8    \\
        Surrogate & 2.509           & 0.1351 & 53.8 & 3.118        & 0.2431 &  30.7 & 1.692          & 0.2135  & 19.9 & 1.694         & 0.2237     &  18.1  \\
        HNA       & \textbf{0.559} & \textbf{0.064} & \textbf{98.1} & \textbf{1.029} & \textbf{0.1412} & \textbf{89.3} & \textbf{0.913} & \textbf{0.113} & \textbf{95.9} & \textbf{0.957}  & \textbf{0.126} & \textbf{94.8}\\ \bottomrule
    \end{tabular}
    }
\end{table}

\subsection{COCO} \label{sec:coco}

The phosphenes produced by \ac{HNA} and the naive encoder for the segmented COCO dataset are shown in Fig.~\ref{fig:coco}. We omit the surrogate results due to its poor perceptual performance on MNIST. Averaged across all $\phi$, \ac{HNA} had a joint loss of 0.713 on the test set and MAE of 0.1408, while the naive encoder had a joint loss of 1.873 and \ac{MAE} of 0.2830.

\begin{figure}[t]
    \centering
    \includegraphics[width=.9\linewidth]{figures/fig-coco.png}
    \caption{Original (\emph{top row}), segmented (\emph{second row}), and reconstructed targets for the COCO dataset, for both \ac{HNA} (\emph{third row}) and naive encoders (\emph{bottom row}). Left to right within each block of 4 images, $\rho$ takes values of 200, 400, 600, 800. Left to right across blocks, $\lambda$ takes values of 250, 750, 1250, 2000. Note that the brightness of the naive method is clipped for display.}
    \label{fig:coco}
\end{figure}

\begin{figure}[t]
    \centering
    \includegraphics[width=0.95\linewidth]{figures/fig-mnist-rho-lam.png}
    \caption{Encoder performance across simulated patients (varying $\rho$ and $\lambda$) on the MNIST dataset. \textbf{\textit{A}}: Target, \ac{HNA} encoder, and naive encoder phosphenes for two example digits. \textbf{\textit{B}}: Heatmaps showing the log joint loss across $\rho$ and $\lambda$ for \ac{HNA} and naive encoders. \textbf{\textit{C}}: T-SNE clusterings on original MNIST targets, \ac{HNA} reconstructed phosphenes, and naive reconstructed phosphenes.}
    \label{fig:mnist-rho-lam}
\end{figure}

\subsection{Modeling Patient-to-Patient Variations} \label{sec:patient-to-patient}
MNIST encoder performance across simulated patients ($\phi$) is shown in Fig.~\ref{fig:mnist-rho-lam}.
Since the surrogate encoder has to be retrained for each patient, comparison is infeasible.
To visualize the effects of changing $\rho$ and $\lambda$ on the produced phosphenes, Fig.~\ref{fig:mnist-rho-lam}A shows the result of encoding two example MNIST digits, both using the naive method and our encoder. As $\lambda$ increases, the naive phosphenes appear increasingly elongated, and as $\rho$ increases, the phosphenes become increasingly large and blurry. The phosphenes from \ac{HNA} are slightly too dim and disconnected at low $\rho$, but are relatively stable across other values of $\rho$ and $\lambda$.

To compare performance across the entire dataset, we computed the average test set loss across the same range of $\rho$ and $\lambda$
(Fig.~\ref{fig:mnist-rho-lam}B).
The encoder performs well across a wide range of simulated patients, with larger loss only at low $\rho$. The naive method performs well only on a limited set of $\phi$, with small $\lambda$ and $\rho \approx 200$. The naive loss was higher than the learned encoder at every simulated point. Random sampling of $\rho$ and $\lambda$ for each image results in a joint loss of 0.921, \ac{MAE} of 0.120, and \ac{RA} of 94.0\% for \ac{HNA}, while the naive encoder results in a joint loss of 3.17, \ac{MAE} of 0.596, and \ac{RA} of 63.6\%.
The same analysis yielded similar results on COCO (Appendix \ref{app:coco_patient}). An analysis across other parameters is presented in Appendix \ref{app:other_modelparams}.

In order for prosthetic vision to be useful, different instances of the same objects would ideally produce similar phosphenes, allowing for consistent perception. To evaluate whether our model achieves this, we cluster the target images and resulting phosphenes using T-SNE \citep{van2008visualizing} shown in Fig.~\ref{fig:mnist-rho-lam}C. The ground truth images form clusters corresponding to the digits 0-9. The phosphenes from our encoder roughly form similar, slightly less separated groupings, whereas the naive phosphenes do not.
To ensure that this was not the result of bad T-SNE hyperparameters, we repeated the clustering across different perplexities and learning rates, obtaining similar or worse results.

\subsection{Joint Perceptual Error Ablation Study}
To show that the joint perceptual metric performs better than any of its individual components, we train models using just the VGG loss (Eq. ~\ref{eq:vgg}), and just MAE (Eq. \ref{eq:mae}). Shown are values for $\rho$=150 and $\lambda$=600. As mentioned previously, encoders trained using just VGG loss fail to converge, thus we pretrain the VGG encoder using \ac{MAE} and smoothing loss, then transition to using only VGG. We do not consider ablating the smoothing term (Eq. \ref{eq:smooth}) because it is simply a regularization term.
Fig.~\ref{fig:ablation} shows the phosphenes produced by \ac{HNA} trained on the joint, VGG-only, and MAE-only loss.

The VGG encoder had a test VGG loss of 4\% lower than the joint model, and the phosphenes it produces have the correct high level perceptual details, but are oversmoothed and blurry. The \ac{MAE} encoder had a final test \ac{MAE} of 9\% lower than the joint model. Despite a lower \ac{MAE}, the reconstructed images appear to be lower quality. The joint model had a \ac{RA} of 99.0\% The VGG encoder had a \ac{RA} of 95.9\%, while the joint model had a \ac{RA} of 77.6\%

\begin{figure}[t]
    \centering
    \includegraphics[width=0.9\linewidth]{figures/fig5_jointloss.png}
    \caption{MNIST images for \ac{HNA} encoders trained using the joint, VGG-only, and MAE-only loss.}
    \label{fig:ablation}
\end{figure}

\section{Discussion} \label{sec:discussion}

\paragraph{Visual Prostheses}
We found that \ac{HNA} is able to produce high-fidelity stimuli from the MNIST and COCO datasets that outperform conventional encoding strategies across all tested conditions. Importantly, \ac{HNA} produces phosphenes that are consistent across representations of the same object (Fig.~\ref{fig:mnist-rho-lam}C), which is critical to allowing prosthesis users to learn to associate certain visual patterns with specific objects.
On the MNIST task, \ac{HNA} produced high quality reconstructions, nearly matching the targets (Figure \ref{fig:mnist-performance}). On the harder COCO task, \ac{HNA} significantly outperformed the naive encoder, but was still unable to capture all of the detail in the images.
In Appendix \ref{app:high-res}, we demonstrate that this is largely due to the implant's limited spatial resolution and not a fundamental limitation of \ac{HNA}.

Another advantage of the \ac{HNA} is that it can be trained to predict stimuli across a wide range of patient-specific parameter values $\phi$, whereas the conventional naive encoder works well only for small values of $\rho$ and $\lambda$.
This may be one reason why the naive encoding strategy has been shown to lead to substantial individual differences in visual outcomes \cite{stingl_interim_2017,peli_testing_2020}.
Our results suggest that stimuli produced with \ac{HNA} may be able to reduce at least some amount of this patient-to-patient variability.

Furthermore, \ac{HNA} also proved superior to a surrogate forward model.
The latter offer an alternative when the forward model is computationally expensive or not differentiable.
Understandably, any inaccuracies in the surrogate model will propagate to the learned encoder during training. However, we observed this phenomenon even when the surrogate performed well individually (see Appendix A), suggesting that even for well trained surrogates, the encoder may still learn to exploit the inexact surrogate instead of learning to invert the true model. We suspect that the surrogate method's inferior performance here compared to \cite{relic_deep_2022} can be explained by our larger stimulus search space. Thus, we cannot currently suggest \ac{HNA} for forward models where a surrogate model must be used, unless the forward model is sufficiently simple or has a small stimulus space.

As expected, both encoders perform well for simulated patients with small axonal current spread ($\lambda$). Interestingly, both encoders do slightly worse with very small radial current spread ($\rho$), suggesting that some amount of phosphene overlap is needed to build shapes. This does align with some previous behavioural results \citep{kasowski_immersive_2022}, and may have implications for the design of future visual prostheses such as \citep{palanker_simultaneous_2022}, which attempt to minimize current spread altogether.

\paragraph{Broader Impacts}

While our work is presented in the context of visual prostheses, the \ac{HNA} framework may apply to any sensory neuroprosthesis where stimulus selection can be informed by a numeric or symbolic forward model.
For example, cochlear implants \cite{wilson_better_1991} could use \ac{HNA} to choose stimuli that result in a desired sound, and spinal cord implants \citep{capogrosso_brainspine_2016} could use \ac{HNA} to find the best way to relay neural signals through a damaged section of the spinal cord.
However, \ac{HNA} might not apply to all neural interfaces, such as systems without a clear neural or perceptual target (\textit{e.g.}, deep brain stimulation for the treatment of Parkinson's \citep{benabid_deep_2003}) or closed-loop systems \citep{bozorgzadeh_neurochemostat_2016, chapman_multifunctional_2018}.

\paragraph{Limitations}
Our end-to-end architecture allows for exact optimization of the forward model, but it also imposes limitations on the stability and speed of training.
For example, we observed that convergence of the encoder required customized learning rate scheduling, careful tuning of metric hyperparameters, and modification of the encoder output layer initialization. Additionally, including the forward model in-the-loop required large amounts of memory and compute time. Notably, these limitations are only applicable during training.
Although this may be application-specific, we expect any complex forward model to encounter similar issues, hindering ease of implementation.

\paragraph{Safety Concerns}
The use of a deep neural network for stimulus encoding does raise safety concerns. Since we cannot examine the process by which stimuli are chosen, it is possible that an \ac{HNA} encoder might produce potentially harmful stimuli that could lead to serious adverse events (\textit{e.g.}, seizures).
However, this concern is mitigated by the fact that most current neuroprostheses are equipped with firmware responsible for verifying that all stimuli stay within FDA-approved safety limits.

\section{Conclusion}

In summary, this paper proposes a hybrid autoencoder structure as a general framework for stimulus optimization in sensory neuroprostheses and, as a proof of concept, demonstrates its utility on the prominent example of visual neuroprostheses, drastically outperforming conventional encoding strategies.
This may prove a promising solution for a variety of neuroprosthetic technologies.

{
\small