Isnt the issue "only" that beam has this code instead of engines?
Assuming beam runner facing api is stable - which must be the case anyway - and that each engine has its integration (flink-beam instead of beam-runners-flink), then this issue disappears by construction.
It also has the advantage to have a better maintenance.
Side note: this is what happent which arquillian, originally the community did all adapters impl then each vendor took it back in house to make it better.
Any way to work in that direction maybe?
The main problem here is that users are forced to upgrade infrastructure to obtain new features in Beam, even when those features actually don't require such changes. As an example, another update to Flink 1.6.0 was proposed (without supporting new functionality in Beam) and we already know that it breaks compatibility (again).
I think that upgrading to a Flink X.Y.0 version isn't a good idea to start with. But besides that, if we want to grow adoption, then we need to focus on stability and delivering improvements to Beam without disrupting users.
In the specific case, ideally the surface of Flink would be backward compatible, allowing us to stick to a minimum version and be able to submit pipelines to Flink endpoints of higher versions. Some work in that direction is underway (like versioning the REST API). FYI, lowest common version is what most projects that depend on Hadoop 2.x follow.
Since Beam with Flink 1.5.x client won't talk to Flink 1.6 and there are code changes required to make it compile, we would need to come up with a more involved strategy to support multiple Flink versions. Till then, I would prefer we favor existing users over short lived experiments, which would mean stick with 1.5.x and not support 1.6.0.
As others have already suggested, I also believe LTS releases is the best we can do as a community right now until portability allows us to decouple what a user writes with and how it runs (the SDK and the SDK environment) from the runner (job service + shared common runner libs + Flink/Spark/Dataflow/Apex/Samza/...).
Dataflow would be highly invested in having the appropriate tooling within Apache Beam to support multiple SDK versions against a runner. This in turn would allow people to use any SDK with any runner and as Robert had mentioned, certain optimizations and features would be disabled depending on the capabilities of the runner and the capabilities of the SDK.
The target audience is people who want to use the latest Beam but do not want to use the latest version of the runner, right?
Runner <-> BeamRunnerSpecificCode <-> CommonBeamRunnerLibs <-> SDK.
(which applies to the job submission as well as execution).
Insomuch as the BeamRunnerSpecificCode uses the public APIs of the runner, hopefully upgrading the runner for minor versions should be a no-op, and we can target the lowest version of the runner that makes sense, allowing the user to link against higher versions at his or her discretion. We should provide built targets that allow this. For major versions, it may make sense to have two distinct BeamRunnerSpecificCode libraries (which may or may not share some common code). I hope these wrappers are not too thick.
There is a tight coupling at the BeamRunnerSpecificCode <-> CommonBeamRunnerLibs layer, but hopefully the bulk of the code lives on the right hand side and can be updated as needed independent of the runner. There may be code of the form "if the runner supports X, do this fast path, otherwise, do this slow path (or reject the pipeline).
I hope the CommonBeamRunnerLibs <-> SDK coupling is fairly loose, to the point that one could use SDKs from different versions of Beam (or even developed outside of Beam) with an older/newer runner. We may need to add versioning to the Fn/Runner/Job API itself to support this. Right now of course we're still in a pre-1.0, rapid-development phase wrt this API.
I totally agree with your points especially the users priorities (stick to the already working version) , and the need to leverage important new features. It is indeed a difficult balance to find .
I can talk for a part I know: for the Spark runner, the aim was to support Dataset native spark API (in place of RDD). For that we needed to upgrade to spark 2.x (and we will probably leverage Beam Row as well).
But such an upgrade is a good amount of work which makes it difficult to commit on a schedule such as "if there is a major new feature on an execution engine that we want to leverage, then the upgrade in Beam will be done within x months".
Regarding your point on portability : decoupling SDK from runner with runner harness and SDK harness might make pipeline authors work easy regarding pipeline maintenance. But, still, if we upgrade runner libs, then the users might have their runner harness not work with their engine version.
If such SDK/runner decoupling is 100% functional, then we could imaging having multiple runner harnesses shipping different versions of the runner libs to solve this problem.
But we would need to support more than one version of the runner libs. We chose not to do this on spark runner.
the binary compatibility of Beam pipelines.
potentially upgrade their Flink cluster with a new version of Beam.
latest features if your use case still works with Beam.
how the other Runner authors intend to do it.
theoretically upgrade the SDK without upgrading the runtime components.
That would allow us to defer the upgrade for a longer time.