How to build a commercial image captioning system?
Thinking Commercially

A commercial solution will need to be able to ascertain, continuously verify, and utilize the best of options for captioning learning models.

Each fairly successful image captioning learning models could be placed in an adapter to provide a common training, optimizing, testing, evaluation, and usage interface.  The hot swappable container-addon mega pattern used in peripheral device installation, RAID, J2EE containers, browsers, and other containment sub-systems can be applied.

System Description

The system acceptance criteria being as follows.


A new model could be added without stopping or starting the system if tauted as successful by others
Any model can be deleted without starting or stopping the system if it has not been performing well
Each possible state of each model added can be given some percentage of the total system resources
Various training processes and hyper parameter tuning processes can be applied to any of those model-state combinations
A-B testing can be done on any model-state combination
Stats can be displayed on any model-state combination any model overall or any state across all added models
The interface for each model requires that its associated model produces an array of caption-reliability pairs with zero ore more pairs
The reliability measure that is included in the pair with the caption is an indication of the model's assessment of appropriateness to the image
The most appropriate caption is picked from the proposals of the various models


States can be idle or any of these.


Training
Testing
Evaluation
Hyper parameter tuning
In use


For example, for the two learning models usable in captioning systems suggested in this question, NIC and Neural Talk 2 we could have a system resource allocation like this:


30% NIC Training
5% NIC Hyper parameter tuning
45% NT2 In use
15% NT2 Evaluation
5% NT2 Hyper parameter tuning


Samples may be pulled from a pool of samples that have been vetted.  That pool may be augmented by real images passing through the system, filtered in accordance with security criteria to avoid external attempts at control.

In the assignment of resources, the sample pool selection criteria must be specified.  If the system is already at 100%, the model-state combinations from which the resources shall be drawn must also be specified.

Handling Multiple Output Options

Since there may be more than one model in use and each model may have zero, one, or multiple caption suggestions for each image, each with a reliability measure, the outputs must be analyzed to provide the best choice to associate with the image being analyzed.  Additional system criteria must cover this process scenario.  For any given image, final evaluation must follow the following general guidelines.


If multiple models produce similar or exactly the same captions, they most be weighted higher in final evaluation.
If the model's reliability is proven in actual use based on feedback from end users, the model's output must be weighed higher accordingly
Re-entrant models (such as reinforcement learning network models) must have access to the end user feedback for additional learning even in in-use state
Clear winners are chosen
Close races are disambiguated through trained functionality
Exact ties are broken through pseudo random index generation


Another artificial network may be placed at the output and appropriate encoding and normalization may be applied before training so that a properly trained network, converged using a quantification of the above additional criteria, can select the best caption from the options for each image.

Phased Development Approach

Phase one of such a system would likely require manual handling of model-state allocation.  Phase two would be semi-automation.  The location of new models would still require expert attention.  Perhaps further in the future a hunt for new models could be automated too.