Spring Cloud Data Flow is a cloud native framework from pivotal that provides a highly productive experience for deploying and managing sophisticated data pipelines consisting of standalone microservices. It facilities creation, orchestration & refactoring of data pipelines with a single programming model for common use cases like data ingest, real time analytics, and data import/export.
Spring Cloud Data Flow is the cloud-native redesign of Spring XD – a project that aimed to simplify the development of Big Data applications. These applications are now autonomous deployment units and they can "natively" run in modern runtimes such as Cloud Foundry, Apache YARN, Apache Mesos, and Kubernetes.
CIGNEX Datamatics has leveraged the below architecture in the existing customer engagements.
The architecture for Spring Cloud Data Flow is divided into a number of distinct components.
The Core domain module comprises the concept of a stream that is a composition of spring-cloud-stream modules in a linear pipeline from a source to a sink, alternatively including processor modules in between. The domain also includes the concept of a task, which may be any process that does not run indefinitely, including Spring Batch jobs.
The Admin provides a REST API and UI. It is an executable Spring Boot application that is profile aware, so that the proper implementation of the Module Deployer SPI will be instantiated based on the environment within which the Admin application itself is running.
The Shell connects to the Admin’s REST API and supports a DSL that simplifies the process of defining a stream and managing its lifecycle.
We will be more than happy to connect and know your business requirements. Contact us to discuss about your Spring Cloud Data Flow Project.