text: define(function () {
  return ["datacollector/UserGuide/Administration/Administration_title.html@@@Administration@@@You can view the directories that the Data Collector uses. You might check the directories being used to access a file in the directory or to increase the amount of available space for a directory...","datacollector/UserGuide/Alerts/RulesAlerts_title.html@@@Rules and Alerts@@@Metric rules and alerts provide notifications about real-time statistics for pipelines...","datacollector/UserGuide/Apx-DataFormats/DataFormat_Title.html@@@Data Formats by Stage@@@The following table lists the data formats supported by each origin. Origin Avro Binary Datagram Delimited Excel JSON Log Protobuf SDC Record Text Whole File XML Amazon S3 Amazon SQS Consumer \u00A0 \u00A0...","datacollector/UserGuide/Apx-GrokPatterns/GrokPatterns_title.html@@@Grok Patterns@@@You can use the grok patterns in this appendix to define the structure of log data...","datacollector/UserGuide/Apx-RegEx/RegEx-Title.html@@@Regular Expressions@@@A regular expression, also known as regex, describes a pattern for a string...","datacollector/UserGuide/Cluster_Mode/AmazonS3Requirements.html@@@Amazon S3 Requirements@@@Cluster EMR batch and cluster batch mode pipelines can process data from Amazon S3...","datacollector/UserGuide/Cluster_Mode/ClusterPipelines.html@@@Cluster Pipeline Overview@@@A cluster pipeline is a pipeline that runs in cluster execution mode. You can run a pipeline in standalone execution mode or cluster execution mode...","datacollector/UserGuide/Cluster_Mode/ClusterPipelines_title.html@@@Cluster Pipelines@@@...","datacollector/UserGuide/Cluster_Mode/HDFSRequirements.html@@@HDFS Requirements@@@Cluster mode pipelines that read from HDFS require the Cloudera distribution of Hadoop (CDH) or Hortonworks Data Platform (HDP). For a list of the supported CDH or HDP versions, see Available Stage...","datacollector/UserGuide/Cluster_Mode/KafkaRequirements.html@@@Kafka Cluster Requirements@@@When you use Kerberos authentication, Data Collector uses the Kerberos principal and keytab to connect to Kafka...","datacollector/UserGuide/Cluster_Mode/MapRRequirements.html@@@MapR Requirements@@@Complete the following steps to configure a cluster pipeline to read from MapR in cluster streaming mode...","datacollector/UserGuide/Cluster_Mode/StageLimitations.html@@@Cluster Pipeline Limitations@@@Please note the following limitations in cluster pipelines: Non-cluster origins - Do not use non-cluster origins in cluster pipelines. For a description of the origins to use, see Cluster Batch and...","datacollector/UserGuide/Configuration/Authentication.html@@@User Authentication@@@Data Collector can authenticate user accounts based on LDAP or files. Best practice is to use LDAP if your organization has it. By default, Data Collector uses file-based authentication...","datacollector/UserGuide/Configuration/Config_title.html@@@Configuration@@@...","datacollector/UserGuide/Configuration/CredentialStores.html@@@Credential Stores@@@Data Collector pipeline stages communicate with external systems to read and write data. Many of these external systems require credentials - user names or passwords - to access the data. When you configure pipeline stages for these external systems, you define the credentials that the stage uses to connect to the system...","datacollector/UserGuide/Configuration/CustomStageLibraries.html@@@Custom Stage Libraries@@@If you develop custom stages, store the stage libraries in a local directory external to the Data Collector installation directory. Use an external directory to enable use of the custom stage...","datacollector/UserGuide/Configuration/DCConfig.html@@@Data Collector Configuration@@@You can edit the Data Collector configuration file, $SDC_CONF/sdc.properties, to configure properties such as the host name and port number and account information for email alerts...","datacollector/UserGuide/Configuration/DCEnvironmentConfig.html@@@Data Collector Environment Configuration@@@Data Collector includes environment variables that define the directories used to store configuration,\n        data, log, and resource files...","datacollector/UserGuide/Configuration/ExternalLibs.html@@@Install External Libraries@@@After you&apos;ve set up the external directory, use the Package Manager within Data Collector to install external libraries...","datacollector/UserGuide/Configuration/HTTP_protocols.html@@@Enabling HTTPS@@@Create a keystore file that includes each private key and public certificate pair signed by the CA. A keystore is used to verify the identity of the client upon a request from an SSL/TLS server...","datacollector/UserGuide/Configuration/JMXMetrics-EnableExternalTools.html@@@Enabling External JMX Tools@@@Data Collector uses JMX metrics to generate the graphical display of the status of a running pipeline. You can provide the same JMX metrics to external tools if desired...","datacollector/UserGuide/Configuration/PublishMetadata.html@@@Working with Data Governance Tools@@@You can configure Data Collector to integrate with data governance tools, giving you visibility into data movement - where the data came from, where it\u2019s going to, and who is interacting with it...","datacollector/UserGuide/Configuration/RolesandPermissions.html@@@Roles and Permissions@@@Data Collector allows you to assign roles and pipeline permissions to users and groups. Roles, such as Creator or Manager, enable users to perform different Data Collector tasks. Each user needs at...","datacollector/UserGuide/Configuration/Vault-Overview.html@@@Accessing Hashicorp Vault Secrets with Vault Functions (Deprecated)@@@Data Collector can use Hashicorp Vault functions to access information, a.k.a. secrets , stored in Hashicorp Vault. However, the Vault functions are now deprecated and will be removed in a future...","datacollector/UserGuide/DPM/AggregatedStatistics.html@@@Pipeline Statistics@@@A Control Hub job defines the pipeline to run and the Data Collectors or Edge Data Collectors\n            (SDC Edge)\n        that run the pipeline. When you start a job, Control Hub remotely runs the pipeline on the group of Data Collectors or Edge Data Collectors.\n        To monitor the job statistics and metrics within Control Hub, you must configure the pipeline to write statistics to Control Hub or to another system...","datacollector/UserGuide/DPM/DPM.html@@@Meet StreamSets Control Hub@@@StreamSets Control HubTM is a central point of control for all of your dataflow pipelines. Control Hub allows teams to build and execute large numbers of complex dataflows at scale...","datacollector/UserGuide/DPM/DPMConfiguration.html@@@Control Hub Configuration File@@@You can customize how a registered Data Collector works with StreamSets Control Hub by editing the Control Hub configuration file, $SDC_CONF/dpm.properties, located in the Data Collector installation...","datacollector/UserGuide/DPM/DPM_title.html@@@StreamSets Control Hub@@@...","datacollector/UserGuide/DPM/OrgUserAccount.html@@@Request a Control Hub Organization and User Account@@@To register a Data Collector with StreamSets Control Hub,\n        you must have a Control Hub user account within an organization...","datacollector/UserGuide/DPM/PipelineManagement.html@@@Pipeline Management with Control Hub@@@After you register a Data Collector with StreamSets Control Hub,\n        you can manage how the pipelines work with Control Hub...","datacollector/UserGuide/DPM/RegisterSDCwithDPM.html@@@Register Data Collector with Control Hub@@@You must register a Data Collector to work with StreamSets Control Hub.\n        When you register a Data Collector, Data Collector generates an authentication token that it uses to issue authenticated requests to Control Hub...","datacollector/UserGuide/DPM/UnregisterSDCwithDPM.html@@@Unregister Data Collector from Control Hub@@@You can unregister a Data Collector from StreamSets Control Hub when you no longer want to use that Data Collector installation with Control Hub...","datacollector/UserGuide/DPM/WorkingWithDPM.html@@@Working with Control Hub@@@Before you can work with StreamSets Control Hub, you must have an Control Hub organization and user account...","datacollector/UserGuide/Data_Formats/Avro.html@@@Avro Data Format@@@Data Collector can read and write Avro data. When reading Avro data, file- and object-based origins, such as the Directory and Amazon S3 origins, generate a Data Collector record for every Avro record...","datacollector/UserGuide/Data_Formats/Binary.html@@@Binary Data Format@@@When reading binary data, origins generate a record with a single byte array field at the root of the record. When the data exceeds the user-defined maximum data size, the origin cannot process the...","datacollector/UserGuide/Data_Formats/DataFormats-Overview.html@@@Data Formats Overview@@@Origins and processors that read files can read uncompressed files, compressed files,\n    archives, and compressed archives...","datacollector/UserGuide/Data_Formats/DataFormats-Title.html@@@Data Formats@@@...","datacollector/UserGuide/Data_Formats/Datagram.html@@@Datagram Data Format@@@When reading datagram data, origins generate a record for every message. The origins can process collectd messages, NetFlow 5 and NetFlow 9 messages , and the following types of syslog messages: RFC...","datacollector/UserGuide/Data_Formats/Delimited.html@@@Delimited Data Format@@@Data Collector can read and write delimited data. Origins that read delimited data generate a record for each delimited line in a file, object, or message. Processors that process delimited data...","datacollector/UserGuide/Data_Formats/Excel.html@@@Excel Data Format@@@You can use file-based origins, such as Directory and Google Cloud Storage , to process Microsoft Excel .xls or .xlsx files. When processing Excel files, an origin creates a record for every row in...","datacollector/UserGuide/Data_Formats/LogFormats.html@@@Log Data Format@@@When you use an origin to read log data, you define the format of the log files to be read...","datacollector/UserGuide/Data_Formats/NetFlow_Overview.html@@@NetFlow Data Processing@@@You can use Data Collector to process NetFlow 5 and NetFlow 9 data. When processing NetFlow 5 data, Data Collector processes flow records based on information in the packet header. Data Collector...","datacollector/UserGuide/Data_Formats/Protobuf-Prerequisites.html@@@Protobuf Data Format Prerequisites@@@Perform the following prerequisites before reading or writing protobuf data...","datacollector/UserGuide/Data_Formats/SDCRecordFormat.html@@@SDC Record Data Format@@@SDC Record is a proprietary data format that Data Collector uses to generate error records. Data Collector can also use the data format to read and write data...","datacollector/UserGuide/Data_Formats/TextCDelim.html@@@Text Data Format with Custom Delimiters@@@By default, the text data format creates records based on line breaks, creating a record for each line of text. You can configure origins to create records based on custom delimiters. Use custom...","datacollector/UserGuide/Data_Formats/WholeFile.html@@@Whole File Data Format@@@You can use the whole file data format to transfer entire files from an origin system to a destination system. With the whole file data format, you can transfer any type of file...","datacollector/UserGuide/Data_Formats/WritingXML.html@@@Writing XML Data@@@When writing XML data, destinations create a valid XML document for each record. The destination requires the record to have a single root field that contains the rest of the record data. When writing...","datacollector/UserGuide/Data_Formats/XMLDFormat.html@@@Reading and Processing XML Data@@@You can parse XML documents from an origin system with an origin enabled for the XML data format. You can also parse XML documents in a field in a Data Collector record with the XML Parser processor...","datacollector/UserGuide/Data_Preview/DataPreview_Title.html@@@Data Preview@@@You can preview data to help build or fine-tune a pipeline. When using Control Hub, you can also use data preview when developing pipeline fragments...","datacollector/UserGuide/Destinations/Aerospike.html@@@Aerospike@@@Supported pipeline types: Data Collector The Aerospike destination writes data to Aerospike. The destination can write to a single Aerospike node or to a cluster of Aerospike nodes. When you configure...","datacollector/UserGuide/Destinations/AmazonS3.html@@@Amazon S3@@@When Data Collector writes data to an Amazon S3 destination, it must pass credentials to Amazon Web Services...","datacollector/UserGuide/Destinations/AzureEventHubProducer.html@@@Azure Event Hub Producer@@@Supported pipeline types: Data Collector The Azure Event Hub Producer writes data to Microsoft Azure Event Hub. To write to Microsoft Azure Data Lake Storage, use the Azure Data Lake Storage...","datacollector/UserGuide/Destinations/AzureIoTHub.html@@@Azure IoT Hub Producer@@@Data Collector functions as a simulated device that sends messages to Azure IoT Hub. Before you configure the Azure IoT Hub Producer destination, register Data Collector as a device in your IoT hub...","datacollector/UserGuide/Destinations/BigQuery.html@@@Google BigQuery@@@The Google BigQuery destination maps fields from records to BigQuery columns in existing tables based on matching names and compatible data types. If needed, the destination converts Data Collector data types to BigQuery data types...","datacollector/UserGuide/Destinations/Bigtable.html@@@Google Bigtable@@@The Google Bigtable destination requires the BoringSSL library. You must download and install the external library so that the Google Bigtable destination can access it...","datacollector/UserGuide/Destinations/Cassandra.html@@@Cassandra@@@Supported pipeline types: Data Collector The Cassandra destination writes data to a Cassandra cluster. When you configure the Cassandra destination, you define connection information and map incoming...","datacollector/UserGuide/Destinations/CoAPClient.html@@@CoAP Client@@@Supported pipeline types: Data Collector Data Collector Edge Constrained Application Protocol (CoAP) is a web transfer protocol designed for machine-to-machine devices. The CoAP Client destination...","datacollector/UserGuide/Destinations/Couchbase.html@@@Couchbase@@@Supported pipeline types: Data Collector The Couchbase destination writes data to Couchbase Server. Couchbase Server is a distributed NoSQL document-oriented database. The destination writes each...","datacollector/UserGuide/Destinations/DataLakeStore.html@@@Azure Data Lake Storage@@@Supported pipeline types: Data Collector The Azure Data Lake Storage destination writes data to Microsoft Azure Data Lake Storage. You can use the Azure Data Lake Storage destination in standalone and...","datacollector/UserGuide/Destinations/Destinations-title.html@@@Destinations@@@...","datacollector/UserGuide/Destinations/Destinations_overview.html@@@Destinations@@@A destination stage represents the target for a pipeline. You can use one or more destinations in a pipeline...","datacollector/UserGuide/Destinations/Elasticsearch.html@@@Elasticsearch@@@Supported pipeline types: Data Collector The Elasticsearch destination writes data to an Elasticsearch cluster, including Elastic Cloud clusters (formerly Found clusters) and Amazon Elasticsearch...","datacollector/UserGuide/Destinations/Flume.html@@@Flume@@@Supported pipeline types: Data Collector The Flume destination writes data to a Flume source. When you write data to Flume, you pass data to a Flume client. The Flume client passes data to hosts based...","datacollector/UserGuide/Destinations/GCS.html@@@Google Cloud Storage@@@Supported pipeline types: Data Collector The Google Cloud Storage destination writes data to objects in Google Cloud Storage. You can use other destinations to write to Google BigQuery , Google...","datacollector/UserGuide/Destinations/HBase.html@@@HBase@@@When you configure the HBase destination, you map fields from records to HBase columns...","datacollector/UserGuide/Destinations/HTTPClient.html@@@HTTP Client@@@Supported pipeline types: Data Collector Data Collector Edge The HTTP Client destination writes data to an HTTP endpoint. The destination sends requests to an HTTP resource URL. Use the HTTP Client...","datacollector/UserGuide/Destinations/HadoopFS-destination.html@@@Hadoop FS@@@The Hadoop FS destination can write data directly to Azure Data Lake Storage using using the ADL protocol provided by Hadoop. The destination can connect using Azure Active Directory Service Principal or refresh-token authentication...","datacollector/UserGuide/Destinations/Hive.html@@@Hive Streaming@@@Supported pipeline types: Data Collector The Hive Streaming destination writes data to Hive tables stored in the ORC (Optimized Row Columnar) file format. The Hive Streaming destination requires Hive...","datacollector/UserGuide/Destinations/HiveMetastore.html@@@Hive Metastore@@@The Hive Metastore destination queries Hive for information and caches the results. When possible, it uses the cache to avoid unnecessary Hive queries...","datacollector/UserGuide/Destinations/InfluxDB.html@@@InfluxDB@@@Supported pipeline types: Data Collector The InfluxDB destination writes data to an InfluxDB database. When you configure the InfluxDB destination, you define connection information, the retention...","datacollector/UserGuide/Destinations/JDBCProducer.html@@@JDBC Producer@@@The JDBC Producer destination can write data to multiple database vendors...","datacollector/UserGuide/Destinations/JMSProducer.html@@@JMS Producer@@@Before you use the JMS Producer, install the JMS drivers for the implementation that you are using...","datacollector/UserGuide/Destinations/KProducer.html@@@Kafka Producer@@@Kafka Producer can write a record to the topic based on an expression. When Kafka Producer evaluates a record, it calculates the expression based on record values and writes the record to the resulting topic...","datacollector/UserGuide/Destinations/KinFirehose.html@@@Kinesis Firehose@@@When Data Collector writes data to a Kinesis Firehose destination, it must pass credentials to Amazon Web Services...","datacollector/UserGuide/Destinations/KinProducer.html@@@Kinesis Producer@@@When Data Collector writes data to a Kinesis Producer destination, it must pass credentials to Amazon Web Services...","datacollector/UserGuide/Destinations/KineticaDB.html@@@KineticaDB@@@Supported pipeline types: Data Collector The KineticaDB destination writes data to a table in a Kinetica cluster using the Kinetica bulk inserter. When you configure the KineticaDB destination, you...","datacollector/UserGuide/Destinations/Kudu.html@@@Kudu@@@Supported pipeline types: Data Collector The Kudu destination writes data to a Kudu cluster. When you configure the Kudu destination, you specify the connection information for one or more Kudu...","datacollector/UserGuide/Destinations/LocalFS.html@@@Local FS@@@Supported pipeline types: Data Collector Use the Local FS destination to write records to files in a local file system. When you configure a Local FS destination, you can define a directory template...","datacollector/UserGuide/Destinations/MQTTPublisher.html@@@MQTT Publisher@@@The MQTT Publisher destination writes messages to a single topic on the MQTT broker. Any MQTT client subscribed to that topic receives the messages. A topic is a string that the broker uses to filter messages for each connected client...","datacollector/UserGuide/Destinations/MapRDB.html@@@MapR DB@@@When you configure the MapR DB destination, you map fields from records to MapR DB columns...","datacollector/UserGuide/Destinations/MapRDBJSON.html@@@MapR DB JSON@@@MapR DB uses a row key to uniquely identify each row in a JSON table. The row key is defined by the _id field of the JSON document stored in the row...","datacollector/UserGuide/Destinations/MapRFS.html@@@MapR FS@@@You can use Kerberos authentication to connect to MapR. When you use Kerberos authentication, the Data Collector uses the Kerberos principal and keytab to connect to MapR. By default, Data Collector uses the user account who started it to connect...","datacollector/UserGuide/Destinations/MapRStreamsProd.html@@@MapR Streams Producer@@@MapR Streams Producer can write a record to the topic based on an expression. When MapR Streams Producer evaluates a record, it calculates the expression based on record values and writes the record to the resulting topic...","datacollector/UserGuide/Destinations/MemSQLLoader.html@@@MemSQL Fast Loader@@@You must install the MemSQL stage library before using the MemSQL Fast Loader destination...","datacollector/UserGuide/Destinations/MongoDB.html@@@MongoDB@@@Supported pipeline types: Data Collector The MongoDB destination writes data to MongoDB. To write to MongoDB, records must include a CRUD operation record header attribute. The CRUD operation header...","datacollector/UserGuide/Destinations/NamedPipe.html@@@Named Pipe@@@Use the mkfifo command to create the named pipe on the same machine where Data Collector is installed...","datacollector/UserGuide/Destinations/PubSubPublisher.html@@@Google Pub/Sub Publisher@@@When configured to use the Google Cloud service account credentials file, the destination checks for the file defined in the destination properties...","datacollector/UserGuide/Destinations/PulsarProducer.html@@@Pulsar Producer@@@If the Pulsar cluster uses security features, you must configure the Pulsar Producer destination to use the same security features to connect to Pulsar...","datacollector/UserGuide/Destinations/RabbitMQ.html@@@RabbitMQ Producer@@@Supported pipeline types: Data Collector RabbitMQ Producer writes AMQP messages to a single RabbitMQ queue. When you configure the destination, you specify the information needed to connect to...","datacollector/UserGuide/Destinations/Redis.html@@@Redis@@@Supported pipeline types: Data Collector The Redis destination writes data to Redis. When you configure the destination, you specify the mode that the destination uses to process records. You also...","datacollector/UserGuide/Destinations/SDC_RPCdest.html@@@SDC RPC@@@In an SDC RPC destination, the RPC connections define where the destination passes data...","datacollector/UserGuide/Destinations/Salesforce.html@@@Salesforce@@@When you configure the Salesforce destination, you map fields in the record to existing fields in the Salesforce object...","datacollector/UserGuide/Destinations/SendResponse.html@@@Send Response to Origin@@@Supported pipeline types: Data Collector The Send Response to Origin destination passes records to the REST Service origin in the pipeline and allows you to specify an HTTP status code for those...","datacollector/UserGuide/Destinations/Snowflake.html@@@Snowflake@@@You must install the Snowflake stage library before using the Snowflake destination.\n        The Snowflake stage library includes the Snowflake JDBC driver that the destination uses to access Snowflake...","datacollector/UserGuide/Destinations/Solr.html@@@Solr@@@The index mode determines how the Solr destination indexes records when writing to Solr.\n  Index mode also determines how the destination handles errors...","datacollector/UserGuide/Destinations/Splunk.html@@@Splunk@@@Splunk requires that the event data and metadata be correctly formatted in the record. If the record is formatted incorrectly, an error occurs and the destination fails to write to Splunk. When you design a pipeline with the Splunk destination, you must ensure that the record sent to the destination uses the required format...","datacollector/UserGuide/Destinations/Syslog.html@@@Syslog@@@A syslog message includes fields such as a timestamp, facility code, severity level,\n        message ID, and the log message itself. You construct the syslog message content by specifying the message values or expressions that evaluate to the message values...","datacollector/UserGuide/Destinations/ToError.html@@@To Error@@@Supported pipeline types: Data Collector Data Collector Edge The To Error destination passes records to the pipeline for error handling. Use the To Error destination to send a stream of records to...","datacollector/UserGuide/Destinations/Trash.html@@@Trash@@@Supported pipeline types: Data Collector Data Collector Edge The Trash destination discards records. Use the Trash destination as a visual representation of records discarded from the pipeline. Or...","datacollector/UserGuide/Destinations/WaveAnalytics.html@@@Einstein Analytics@@@In previous releases, you could configure the destination to use an Einstein Analytics dataflow to combine multiple datasets together. However, using dataflows is now deprecated and will be removed in a future release. We recommend configuring the destination to use the append operation to combine data into a single dataset...","datacollector/UserGuide/Destinations/WebSocketClient.html@@@WebSocket Client@@@Supported pipeline types: Data Collector Data Collector Edge The WebSocket Client destination writes data to a WebSocket endpoint. Use the destination to send data to a WebSocket resource URL. The...","datacollector/UserGuide/Edge_Mode/DownloadPipelines.html@@@Downloading Pipelines from SDC Edge@@@When SDC Edge is running and is accessible by the Data Collector machine, you can download edge pipelines from SDC Edge into Data Collector.\n        When you download pipelines from SDC Edge, you download all edge pipelines deployed to that SDC Edge...","datacollector/UserGuide/Edge_Mode/EdgePipelineTypes.html@@@Design Edge Pipelines@@@Edge pipelines run in edge execution mode. You design edge pipelines in Data Collector...","datacollector/UserGuide/Edge_Mode/EdgePipelines_Deploy.html@@@Deploy Pipelines to SDC Edge@@@After designing edge pipelines in Data Collector, you deploy the edge pipelines to SDC Edge installed on an edge device. You run the edge pipelines on SDC Edge...","datacollector/UserGuide/Edge_Mode/EdgePipelines_Manage.html@@@Manage Pipelines on SDC Edge@@@After designing edge pipelines in Data Collector and then deploying the edge pipelines to SDC Edge, you can manage the pipelines on SDC Edge.\n        Managing edge pipelines includes previewing, validating, starting, stopping, and monitoring the pipelines as well as resetting the origin for the pipelines...","datacollector/UserGuide/Edge_Mode/EdgePipelines_Overview.html@@@Meet StreamSets Data Collector Edge@@@StreamSets Data Collector EdgeTM (SDC Edge) is a lightweight execution agent without a UI that runs pipelines on edge devices with limited resources. Use SDC Edge to read data from an edge device or to receive data from another pipeline and then act on that data to control an edge device...","datacollector/UserGuide/Edge_Mode/EdgePipelines_title.html@@@StreamSets Data Collector Edge@@@...","datacollector/UserGuide/Edge_Mode/GettingStartedSamples.html@@@Getting Started with SDC Edge@@@Data Collector Edge (SDC Edge)\n        includes several sample pipelines that make it easy to get started. You simply import one of the sample edge pipelines, create the appropriate Data Collector receiving pipeline, download and install SDC Edge on the edge device, and then run the sample edge pipeline...","datacollector/UserGuide/Edge_Mode/SDCEInstall.html@@@Install SDC Edge@@@Download and install SDC Edge on each edge device where you want to run edge pipelines...","datacollector/UserGuide/Edge_Mode/SDCReceivingPipelines.html@@@Design Data Collector Receiving Pipelines@@@Data Collector receiving pipelines run in the standalone execution mode. You design and run receiving pipelines in Data Collector...","datacollector/UserGuide/Edge_Mode/SDCeAdminister.html@@@Administer SDC Edge@@@Administering SDC Edge involves configuring, starting, shutting down, and viewing logs for the agent. When using StreamSets Control Hub, you can also use the SDC Edge command line interface to register SDC Edge with Control Hub...","datacollector/UserGuide/Edge_Mode/SupportedPlatforms.html@@@Supported Platforms@@@Install Data Collector Edge\n            (SDC Edge)\n        on each edge device where you want to run edge pipelines...","datacollector/UserGuide/Event_Handling/EventFramework-Title.html@@@Dataflow Triggers@@@Dataflow triggers are instructions for the event framework to kick off tasks in response to events that occur in the pipeline. For example, you can use dataflow triggers to start a MapReduce job after...","datacollector/UserGuide/Executors/AmazonS3.html@@@Amazon S3 Executor@@@When Data Collector uses the Amazon S3 executor, it must pass credentials to Amazon Web Services...","datacollector/UserGuide/Executors/Databricks.html@@@Databricks Executor@@@The Databricks executor can generate events that you can use in an event stream. When you enable event generation, the executor generates events each time it starts a Databricks job...","datacollector/UserGuide/Executors/Email.html@@@Email Executor@@@The Email executor sends the configured email to the specified recipients upon receiving an event. You can also configure the executor to send an email based on a condition, such as the arrival of a...","datacollector/UserGuide/Executors/Executors-overview.html@@@Executors@@@An executor stage triggers a task when it receives an event. Executors do not write or store events. Use executors as part of a dataflow trigger in an event stream to perform event-driven...","datacollector/UserGuide/Executors/Executors-title.html@@@Executors@@@...","datacollector/UserGuide/Executors/HDFSMetadata.html@@@HDFS File Metadata Executor@@@The HDFS File Metadata executor changes file metadata, creates an empty file, or removes a file or directory in HDFS or a local file system each time it receives an event. You cannot perform multiple...","datacollector/UserGuide/Executors/HiveQuery.html@@@Hive Query Executor@@@The Hive Query executor connects to Hive or Impala and performs one or more user-defined Hive or Impala queries each time it receives an event record. Use the Hive Query executor as part of an event...","datacollector/UserGuide/Executors/JDBCQuery.html@@@JDBC Query Executor@@@The JDBC Query executor can read database data from multiple database vendors...","datacollector/UserGuide/Executors/MapRFSFileMeta.html@@@MapR FS File Metadata Executor@@@The MapR FS File Metadata executor changes file metadata, creates an empty file, or removes a file or directory in MapR FS each time it receives an event. You cannot perform multiple tasks in the same...","datacollector/UserGuide/Executors/MapReduce.html@@@MapReduce Executor@@@The MapReduce executor includes two predefined jobs: Avro to ORC and Avro to Parquet...","datacollector/UserGuide/Executors/PipelineFinisher.html@@@Pipeline Finisher Executor@@@When it receives an event, the Pipeline Finisher executor stops a pipeline and transitions it to a Finished state. This allows the pipeline to complete all expected processing before stopping. Use the...","datacollector/UserGuide/Executors/Shell.html@@@Shell Executor@@@When Data Collector is registered with Control Hub, you can configure Data Collector to use an abbreviated version of the Control Hub user ID for shell impersonation mode...","datacollector/UserGuide/Executors/Spark.html@@@Spark Executor@@@The Spark executor starts a Spark application each time it receives an event. You can use the Spark executor with Spark on YARN. The executor is not compatible with Spark on Mesos at this time. Use...","datacollector/UserGuide/Expression_Language/Constants.html@@@Constants@@@The expression language provides constants for use in expressions...","datacollector/UserGuide/Expression_Language/DateTimeVariables.html@@@Datetime Variables@@@The expression language provides datetime variables for use in expressions...","datacollector/UserGuide/Expression_Language/ExpressionLanguage_overview.html@@@Expression Language@@@Use the following information and tips when you invoke expression completion...","datacollector/UserGuide/Expression_Language/ExpressionLanguage_title.html@@@Expression Language@@@...","datacollector/UserGuide/Expression_Language/Functions.html@@@Functions@@@Use record functions to determine information about a record, such as the stage that created it or whether a field exists in the record...","datacollector/UserGuide/Expression_Language/Literals.html@@@Literals@@@The expression language includes the following literals...","datacollector/UserGuide/Expression_Language/Operators.html@@@Operators@@@The precedence of operators highest to lowest, left to right is as follows...","datacollector/UserGuide/Expression_Language/ReservedWords.html@@@Reserved Words@@@The following words are reserved for the expression language and should not be used as identifiers...","datacollector/UserGuide/Getting_Started/GettingStarted_Title.html@@@Getting Started@@@StreamSets Data CollectorTM is a lightweight, powerful design and execution engine that streams data in real time. Use Data Collector to route and process data in your data streams...","datacollector/UserGuide/Glossary/Glossary_title.html@@@Glossary@@@batch A set of records that passes through a pipeline. Data Collector processes data in batches. CDC-enabled origin An origin that can process changed data and place CRUD operation information in the...","datacollector/UserGuide/Hive_Drift_Solution/HiveDriftSolution_title.html@@@Drift Synchronization Solution for Hive@@@Now to process the metadata records - and to automatically create and update tables in Hive - you need the Hive Metastore destination...","datacollector/UserGuide/Installation/AddtionalStageLibs.html@@@Install Additional Stage Libraries@@@Install additional stage libraries to use stages that are not included in the core RPM or core tarball installation of Data Collector . This is an optional step, but generally you&apos;ll want to install...","datacollector/UserGuide/Installation/CMInstall-Overview.html@@@Installation with Cloudera Manager@@@When administering Data Collector with Cloudera Manager, configure all Data Collector configuration properties and environment variables through Cloudera Manager...","datacollector/UserGuide/Installation/CloudInstall.html@@@Installation with Cloud Service Providers@@@You can install the full Data Collector on a CentOS 7.x virtual machine hosted on Microsoft Azure...","datacollector/UserGuide/Installation/CoreInstall_Overview.html@@@Core Installation@@@You can download and install a core version of Data Collector, and then install individual stage libraries as needed. Use the core installation to install only the stage libraries that you want to use. The core installation allows Data Collector to use less disk space...","datacollector/UserGuide/Installation/CreateAnotherDC.html@@@Creating Another Data Collector Instance@@@You can create another instance of a Data Collector tarball or RPM installation on the same machine with the create-dc command.\n        The additional Data Collector instance uses the same configuration as the original Data Collector instance. You can modify the configuration properties as needed...","datacollector/UserGuide/Installation/FullInstall_ServiceStart.html@@@Full Installation and Launch (Service Start)@@@You can install the Data Collector RPM package and start it as a service on CentOS, Oracle Linux, or Red Hat Enterprise Linux...","datacollector/UserGuide/Installation/Install_title.html@@@Installation@@@...","datacollector/UserGuide/Installation/InstallationAndConfig.html@@@Installation@@@You can install Data Collector and start it manually or run it as a service...","datacollector/UserGuide/Installation/Installing_the_DC-Docker.html@@@Run Data Collector from Docker@@@You can run the Data Collector image from Docker Hub...","datacollector/UserGuide/Installation/Installing_the_DC.html@@@Full Installation and Launch (Manual Start)@@@You can install the full Data Collector tarball and start it manually on all supported operating systems...","datacollector/UserGuide/Installation/MapR-Prerequisites.html@@@MapR Prerequisites@@@Due to licensing restrictions, StreamSets cannot distribute MapR libraries with Data Collector. As a result, you must perform additional steps to enable the Data Collector machine to connect to MapR. Data Collector does not display MapR origins and destinations in stage library lists nor the MapR Streams statistics aggregator in the pipeline properties until you perform these prerequisites...","datacollector/UserGuide/Installation/Uninstall.html@@@Uninstallation@@@You can uninstall a Data Collector instance that was installed from the tarball, from the RPM package, or from Cloudera Manager. To uninstall a Data Collector instance that was installed from the...","datacollector/UserGuide/JDBC_DriftSolution/JDBC_DriftSyncSolution_title.html@@@Drift Synchronization Solution for PostgreSQL@@@The Drift Synchronization Solution for PostgreSQL detects drift in incoming data and automatically creates or alters corresponding PostgreSQL tables as needed before the data is written. For example...","datacollector/UserGuide/Microservice/Microservice_Title.html@@@Microservice Pipelines@@@A microservice pipeline is a pipeline that creates a fine-grained service to perform a specific task...","datacollector/UserGuide/Multithreaded_Pipelines/MultithreadedPipelines.html@@@Multithreaded Pipelines@@@A multithreaded pipeline is a Data Collector pipeline with an origin that supports parallel execution, enabling one pipeline to run in multiple threads. Not valid in Data Collector Edge pipelines...","datacollector/UserGuide/Origins/AmazonS3.html@@@Amazon S3@@@When Data Collector reads data from an Amazon S3 origin, it must pass credentials to Amazon Web Services...","datacollector/UserGuide/Origins/AmazonSQS.html@@@Amazon SQS Consumer@@@When Data Collector reads data from an Amazon SQS Consumer origin, it must pass credentials to Amazon Simple Queue Services...","datacollector/UserGuide/Origins/AzureEventHub.html@@@Azure IoT/Event Hub Consumer@@@Supported pipeline types: Data Collector The Azure IoT/Event Hub Consumer origin reads data from Microsoft Azure Event Hub. The origin can use multiple threads to enable parallel processing of data...","datacollector/UserGuide/Origins/BigQuery.html@@@Google BigQuery@@@When configured to use the Google Cloud service account credentials file, the origin checks for the file defined in the origin properties...","datacollector/UserGuide/Origins/CoAPServer.html@@@CoAP Server@@@Before you run a pipeline with the CoAP Server origin, configure the CoAP clients to send data to the CoAP Server listening port and resource...","datacollector/UserGuide/Origins/Directory.html@@@Directory@@@Use a file name pattern to define the files that the Directory origin processes. You can use either a glob pattern or a regular expression to define the file name pattern...","datacollector/UserGuide/Origins/Elasticsearch.html@@@Elasticsearch@@@Define the query that the origin uses to return data from Elasticsearch. You can define any valid Elasticsearch query...","datacollector/UserGuide/Origins/FileTail.html@@@File Tail@@@File Tail processes the active file and archived files based on how the source server generates files. When you specify the naming convention for archived files, File Tail determines the file generation method and processes the data accordingly...","datacollector/UserGuide/Origins/GCS.html@@@Google Cloud Storage@@@Supported pipeline types: Data Collector The Google Cloud Storage origin reads objects stored in Google Cloud Storage. The objects must be fully written and reside in a single bucket. The object names...","datacollector/UserGuide/Origins/HDFSStandalone.html@@@Hadoop FS Standalone@@@Supported pipeline types: Data Collector The Hadoop FS Standalone origin reads files in HDFS. You can also use the origin to read from Azure Data Lake Storage or Azure HDInsight. The files to be...","datacollector/UserGuide/Origins/HTTPClient.html@@@HTTP Client@@@When using page number or offset number pagination, the API of the HTTP client typically requires that you include a page or offset parameter at the end of the resource URL. The parameter determines the next page or offset of data to request...","datacollector/UserGuide/Origins/HTTPServer.html@@@HTTP Server@@@Configure the HTTP clients to send data to the HTTP Server listening port...","datacollector/UserGuide/Origins/HTTPtoKafka.html@@@HTTP to Kafka (Deprecated)@@@The HTTP to Kafka origin listens on an HTTP endpoint and writes the contents of all authorized HTTP POST requests directly to Kafka. However, the HTTP to Kafka origin is now deprecated and will be...","datacollector/UserGuide/Origins/HadoopFS-origin.html@@@Hadoop FS@@@The Hadoop FS origin included in a cluster batch or cluster EMR batch pipeline allows you to read from Amazon S3...","datacollector/UserGuide/Origins/JDBCConsumer.html@@@JDBC Query Consumer@@@The JDBC Query Consumer origin can read database data from multiple database vendors...","datacollector/UserGuide/Origins/JMS.html@@@JMS Consumer@@@Before you use the JMS Consumer, install the JMS drivers for the implementation that you are using...","datacollector/UserGuide/Origins/KConsumer.html@@@Kafka Consumer@@@The first time that a Kafka Consumer origin identified by a consumer group receives messages from a topic, an offset entry is created for that consumer group and topic. The offset entry is created in ZooKeeper or Kafka, depending on your Kafka version and broker configuration...","datacollector/UserGuide/Origins/KafkaMultiConsumer.html@@@Kafka Multitopic Consumer@@@The first time that a Kafka Multitopic Consumer origin identified by a consumer group receives messages from a topic, an offset entry is created for that consumer group and topic.\n    The offset entry is created in Kafka...","datacollector/UserGuide/Origins/KinConsumer.html@@@Kinesis Consumer@@@When Data Collector reads data from a Kinesis Consumer origin, it must pass credentials to Amazon Web Services...","datacollector/UserGuide/Origins/MQTTSubscriber.html@@@MQTT Subscriber@@@The MQTT Subscriber origin reads messages from one or more topics on the MQTT broker. A topic is a string that the broker uses to filter messages for each connected client...","datacollector/UserGuide/Origins/MapRDBJSON.html@@@MapR DB JSON@@@When the origin converts a JSON document into a record, it includes the _id field of the JSON document in the record. If needed, you can use the Field Remover processor in the pipeline to remove the _id field...","datacollector/UserGuide/Origins/MapRFS.html@@@MapR FS@@@You can use Kerberos authentication to connect to MapR. When you use Kerberos authentication, Data Collector uses the Kerberos principal and keytab to connect to MapR. By default, Data Collector uses the user account who started it to connect...","datacollector/UserGuide/Origins/MapRFSStandalone.html@@@MapR FS Standalone@@@Use a file name pattern to define the files that the MapR FS Standalone origin processes. You can use either a glob pattern or a regular expression to define the file name pattern...","datacollector/UserGuide/Origins/MapRStreamsCons.html@@@MapR Streams Consumer@@@When you start a pipeline for the first time, the MapR Streams Consumer becomes a new consumer group for the topic. It reads only incoming data, processing data from all partitions, and ignores any existing data in the topic by default...","datacollector/UserGuide/Origins/MapRStreamsMultiConsumer.html@@@MapR Multitopic Streams Consumer@@@You can add custom configuration properties to the MapR Multitopic Streams Consumer. You can use any MapR or Kafka property supported by MapR Streams. For more information, see the MapR Streams documentation...","datacollector/UserGuide/Origins/MapRdbCDC.html@@@MapR DB CDC@@@Supported pipeline types: Data Collector The MapR DB CDC origin reads changed data from MapR DB that has been written to MapR Streams. The origin can use multiple threads to enable parallel processing...","datacollector/UserGuide/Origins/MongoDB.html@@@MongoDB@@@Supported pipeline types: Data Collector The MongoDB origin reads data from MongoDB. Data Collector generates a record for every MongoDB document. To read change data capture information from the...","datacollector/UserGuide/Origins/MongoDBOplog.html@@@MongoDB Oplog@@@Supported pipeline types: Data Collector The MongoDB Oplog origin reads entries from MongoDB Oplog. MongoDB stores information about changes to the database in a local capped collection called an...","datacollector/UserGuide/Origins/MultiTableJDBCConsumer.html@@@JDBC Multitable Consumer@@@The JDBC Multitable Consumer origin can read database data from multiple database vendors...","datacollector/UserGuide/Origins/MySQLBinaryLog.html@@@MySQL Binary Log@@@The MySQL Binary Log origin can process binary logs from a MySQL server configured to use row-based logging...","datacollector/UserGuide/Origins/OPCUAClient.html@@@OPC UA Client@@@Supported pipeline types: Data Collector The OPC UA Client origin processes data from an OPC UA server. The OPC UA Client origin can poll the server at regular intervals, returning the latest data...","datacollector/UserGuide/Origins/Omniture.html@@@Omniture@@@Supported pipeline types: Data Collector The Omniture origin processes JSON website usage reports generated by the Omniture reporting APIs. Omniture is also known as the Adobe Marketing Cloud. When...","datacollector/UserGuide/Origins/OracleBulk.html@@@Oracle Bulkload@@@You must install the Oracle Bulkload stage library before using the Oracle Bulkload origin...","datacollector/UserGuide/Origins/OracleCDC.html@@@Oracle CDC Client@@@The Oracle CDC Client origin provides an alternate PEG parser that you can try when concerned about pipeline performance...","datacollector/UserGuide/Origins/Origins_overview.html@@@Origins@@@An origin stage represents the source for the pipeline. You can use a single origin stage in a pipeline...","datacollector/UserGuide/Origins/Origins_title.html@@@Origins@@@...","datacollector/UserGuide/Origins/PostgreSQL.html@@@PostgreSQL CDC Client@@@Supported pipeline types: Data Collector The PostgreSQL CDC Client origin processes Write-Ahead Logging (WAL) data to generate change data capture records for a PostgreSQL database. Use the PostgreSQL...","datacollector/UserGuide/Origins/PubSub.html@@@Google Pub/Sub Subscriber@@@When configured to use the Google Cloud service account credentials file, the origin checks for the file defined in the origin properties...","datacollector/UserGuide/Origins/PulsarConsumer.html@@@Pulsar Consumer@@@The Pulsar Consumer origin can subscribe to a single topic or to multiple topics. In either case, the origin uses one thread for the read...","datacollector/UserGuide/Origins/RESTService.html@@@REST Service@@@Configure the REST API clients to include the REST Service application ID in each request...","datacollector/UserGuide/Origins/RabbitMQ.html@@@RabbitMQ Consumer@@@RabbitMQ Consumer reads messages from a specified queue...","datacollector/UserGuide/Origins/Redis.html@@@Redis Consumer@@@You can specify channels and patterns to define the data that the Redis Consumer origin processes...","datacollector/UserGuide/Origins/SDCRPCtoKafka.html@@@SDC RPC to Kafka (Deprecated)@@@You can specify the maximum number of requests the SDC RPC to Kafka origin handles at one time...","datacollector/UserGuide/Origins/SDC_RPCorigin.html@@@SDC RPC@@@Supported pipeline types: Data Collector The SDC RPC origin enables connectivity between two SDC RPC pipelines. The SDC RPC origin reads data passed from an SDC RPC destination. Use the SDC RPC origin...","datacollector/UserGuide/Origins/SFTP.html@@@SFTP/FTP Client@@@Use a file name pattern to define the files that the SFTP/FTP Client origin processes. You can use either a glob pattern or a regular expression to define the file name pattern...","datacollector/UserGuide/Origins/SQLServerCDC.html@@@SQL Server CDC Client@@@Supported pipeline types: Data Collector The SQL Server CDC Client origin processes data in Microsoft SQL Server change data capture (CDC) tables. The origin fetches changes in time windows and uses...","datacollector/UserGuide/Origins/SQLServerChange.html@@@SQL Server Change Tracking@@@You can define the initial order that the origin uses to read the tables...","datacollector/UserGuide/Origins/Salesforce.html@@@Salesforce@@@The Salesforce origin can execute a query to read existing data from Salesforce. Use the Salesforce Object Query Language (SOQL) to write the query...","datacollector/UserGuide/Origins/SystemMetrics.html@@@System Metrics@@@The System Metrics origin uses the psutil package for the Go programming language (or Golang) to collect system metrics...","datacollector/UserGuide/Origins/TCPServer.html@@@TCP Server@@@Supported pipeline types: Data Collector The TCP Server origin listens at the specified port numbers, establishes TCP sessions with clients that initiate TCP connections, and then processes the...","datacollector/UserGuide/Origins/Teradata.html@@@Teradata Consumer@@@You must install the Teradata stage library before using the Teradata Consumer origin...","datacollector/UserGuide/Origins/UDP.html@@@UDP Source@@@Supported pipeline types: Data Collector The UDP Source origin reads messages from one or more UDP ports. To use multiple threads for pipeline processing, use the UDP Multithreaded Source . For a...","datacollector/UserGuide/Origins/UDPMulti.html@@@UDP Multithreaded Source@@@Supported pipeline types: Data Collector The UDP Multithreaded Source origin reads messages from one or more UDP ports. The origin can create multiple worker threads to enable parallel processing in a...","datacollector/UserGuide/Origins/UDPtoKafka.html@@@UDP to Kafka (Deprecated)@@@When you use a UDP to Kafka origin in a pipeline, connect the origin to a Trash destination...","datacollector/UserGuide/Origins/WebSocketClient.html@@@WebSocket Client@@@Supported pipeline types: Data Collector Data Collector Edge The WebSocket Client origin reads data from a WebSocket server endpoint. Use the origin to read data from a WebSocket resource URL. The...","datacollector/UserGuide/Origins/WebSocketServer.html@@@WebSocket Server@@@Configure the WebSocket clients to send data to the WebSocket Server listening port...","datacollector/UserGuide/Origins/WindowsLog.html@@@Windows Event Log@@@Supported pipeline types: Data Collector Data Collector Edge The Windows Event Log origin reads data from a Microsoft Windows event log located on a Windows machine. The origin generates a record for...","datacollector/UserGuide/Origins/gRPCClient.html@@@gRPC Client@@@Before the gRPC Client origin can process data from a gRPC server, you must enable reflection for the server...","datacollector/UserGuide/Pipeline_Configuration/ConfiguringAPipeline.html@@@Configuring a Pipeline@@@Configure a pipeline to define the stream of data. After you configure the pipeline,\n        you can start the pipeline...","datacollector/UserGuide/Pipeline_Configuration/DataCollectorUI-Config.html@@@Data Collector UI - Edit Mode@@@The following image shows the Data Collector UI when you configure a pipeline: Area / Icon Name Description 1 Pipeline canvas Displays the pipeline. Use to configure the pipeline data flow. 2 Pipeline...","datacollector/UserGuide/Pipeline_Configuration/EventGeneration.html@@@Event Generation@@@The event framework generates pipeline events for Data Collector standalone pipelines when the pipeline starts and stops. Not available in Data Collector Edge pipelines. In a Data Collector standalone...","datacollector/UserGuide/Pipeline_Configuration/Expressions.html@@@Expression Configuration@@@Precede all expressions with a dollar sign and enclose them with curly brackets, as follows: ${&lt;expression&gt;}...","datacollector/UserGuide/Pipeline_Configuration/Notifications.html@@@Notifications@@@You can configure a pipeline to send an email or webhook when the pipeline changes to specified states. For example, you might send an email when someone manually stops the pipeline, causing it to...","datacollector/UserGuide/Pipeline_Configuration/PipelineConfiguration_title.html@@@Pipeline Configuration@@@...","datacollector/UserGuide/Pipeline_Configuration/PipelineRateLimit.html@@@Rate Limit@@@You can limit the rate at which a Data Collector pipeline processes records by defining the maximum number of records that the pipeline can read in a second...","datacollector/UserGuide/Pipeline_Configuration/ProductIcons_Doc.html@@@Pipeline Types and Icons in Documentation@@@In Data Collector, you can configure pipelines that are run by Data Collector and pipelines that are run by Data Collector Edge...","datacollector/UserGuide/Pipeline_Configuration/Retry.html@@@Retrying the Pipeline@@@By default, when Data Collector encounters a stage-level error that might cause a standalone pipeline to fail, it retries the pipeline. That is, it waits a period of time, and then tries again to run the pipeline...","datacollector/UserGuide/Pipeline_Configuration/RuntimeValues.html@@@Runtime Values@@@Runtime values are values that you define outside of the pipeline and use for stage and pipeline properties. You can change the values for each pipeline run without having to edit the pipeline...","datacollector/UserGuide/Pipeline_Configuration/SSL-TLS.html@@@SSL/TLS Configuration@@@Some stages allow you use SSL/TLS to securely connect to the external system. When you enable TLS, you can generally configure properties on the TLS tab of the stage. The properties that are available...","datacollector/UserGuide/Pipeline_Configuration/SimpleBulkEdit.html@@@Simple and Bulk Edit Mode@@@Some pipeline and pipeline stage properties include an Add icon ( ) where you add additional configurations. You can add the configurations in simple or bulk edit mode. By default, each property uses...","datacollector/UserGuide/Pipeline_Configuration/Validation.html@@@Implicit and Explicit Validation@@@Data Collector performs two types of validation: Implicit validation Implicit validation occurs by default as the Data Collector UI saves your changes. Implicit validation lists missing or incomplete...","datacollector/UserGuide/Pipeline_Configuration/Webhooks.html@@@Webhooks@@@You can configure a pipeline to use webhooks. Not available in Data Collector Edge pipelines. A webhook is a user-defined HTTP callback - an HTTP request that the pipeline sends automatically when...","datacollector/UserGuide/Pipeline_Design/CDC-Overview.html@@@Processing Changed Data@@@Certain stages enable you to easily process change capture data (CDC) or transactional data in a pipeline. CDC-enabled origins can read change capture data. Some exclusively read change capture data...","datacollector/UserGuide/Pipeline_Design/ControlCharacters.html@@@Control Character Removal@@@You can use several stages to remove control characters - such as escape or end-of-transmission characters - from data. Remove control characters to avoid creating invalid records...","datacollector/UserGuide/Pipeline_Design/DatainMotion.html@@@Data in Motion@@@Data passes through the pipeline in batches. This is how it works...","datacollector/UserGuide/Pipeline_Design/DesigningDataFlow.html@@@Designing the Data Flow@@@You can branch and merge streams in the pipeline...","datacollector/UserGuide/Pipeline_Design/DevStages.html@@@Development Stages@@@You can use several development stages to help develop and test pipelines. Note: Do not use development stages in production pipelines. You can use the following stages when developing or testing...","datacollector/UserGuide/Pipeline_Design/DroppingUnwantedRecords.html@@@Dropping Unwanted Records@@@You can drop records from the pipeline at each stage by defining required fields or preconditions for a record to enter a stage...","datacollector/UserGuide/Pipeline_Design/ErrorHandling.html@@@Error Record Handling@@@You can configure error record handling at a stage level and at a pipeline level. You can also specify the version of the record to use as the basis for the error record. When an error occurs as a...","datacollector/UserGuide/Pipeline_Design/FieldAttributes.html@@@Field Attributes@@@Field attributes are attributes that provide additional information about each field that you can use in pipeline logic, as needed...","datacollector/UserGuide/Pipeline_Design/PipelineDesign_title.html@@@Pipeline Concepts and Design@@@...","datacollector/UserGuide/Pipeline_Design/RecordHeaderAttributes.html@@@Record Header Attributes@@@Record header attributes are attributes in record headers that you can use in pipeline logic, as needed. Some stages create record header attributes for a particular purpose. For example, CDC-enabled...","datacollector/UserGuide/Pipeline_Design/SDCRecordFormat.html@@@SDC Record Data Format@@@SDC Record is a proprietary data format that Data Collector uses to generate error records. Data Collector can also use the data format to read and write data...","datacollector/UserGuide/Pipeline_Design/TechPreview.html@@@Technology Preview Functionality@@@Data Collector includes certain new features and stages with the Technology Preview designation. Technology Preview functionality is available for use in development and testing, but is not meant for use in production...","datacollector/UserGuide/Pipeline_Design/TestOrigin.html@@@Test Origin for Preview@@@A test origin can provide test data for data preview to aid in pipeline development.\n        In Control Hub, you can also use test origins when developing pipeline fragments. Test origins are not used when running a pipeline...","datacollector/UserGuide/Pipeline_Design/What_isa_Pipeline.html@@@What is a Pipeline?@@@A pipeline describes the flow of data from the origin system to destination systems and defines how to transform the data along the way...","datacollector/UserGuide/Pipeline_Fragments/DataPreview.html@@@Data Preview@@@You can use data preview to help develop or test a pipeline fragment...","datacollector/UserGuide/Pipeline_Maintenance/PipelineMaintenance_title.html@@@Pipeline Maintenance@@@If you defined runtime parameters for a pipeline, you can specify the parameter values to use when you start the pipeline...","datacollector/UserGuide/Pipeline_Maintenance/PipelineStates-Examples.html@@@State Transition Examples@@@Here are some examples of how pipelines can move through states: Starting a pipeline When you successfully start a pipeline for the first time, a pipeline transitions through the following states...","datacollector/UserGuide/Pipeline_Maintenance/PipelineStates-Understanding.html@@@Understanding Pipeline States@@@A pipeline state is the current condition of the pipeline, such as &quot;running&quot; or &quot;stopped&quot;. The pipeline state can display in the All Pipelines list. The state of a pipeline can also appear in the Data...","datacollector/UserGuide/Pipeline_Monitoring/PipelineMonitoring_title.html@@@Pipeline Monitoring@@@When the Data Collector runs a pipeline, you can view real-time statistics about the pipeline, examine a sample of the data being processed, and create rules and alerts...","datacollector/UserGuide/Processors/Aggregator.html@@@Windowing Aggregator@@@Supported pipeline types: Data Collector The Windowing Aggregator processor performs one or more aggregations within a window of time. The Windowing Aggregator processor displays the results in...","datacollector/UserGuide/Processors/Base64Decoder.html@@@Base64 Field Decoder@@@Supported pipeline types: Data Collector The Base64 Field Decoder decodes Base64 encoded data to binary data. Use the processor to decode Base64 encoded data before evaluating data in the field. When...","datacollector/UserGuide/Processors/Base64Encoder.html@@@Base64 Field Encoder@@@Supported pipeline types: Data Collector The Base64 Field Encoder encodes binary data using Base64. Use the processor to encode binary data that must be sent over channels that expect ASCII data. For...","datacollector/UserGuide/Processors/DataGenerator.html@@@Data Generator@@@When you use the Data Generator processor, you specify the target field for the serialized record...","datacollector/UserGuide/Processors/DataParser.html@@@Data Parser@@@Supported pipeline types: Data Collector The Data Parser processor allows you to parse supported data formats embedded in a field. You can parse NetFlow embedded in a byte array field or syslog...","datacollector/UserGuide/Processors/DatabricksML.html@@@Databricks ML Evaluator@@@Supported pipeline types: Data Collector The Databricks ML Evaluator processor uses a machine learning model exported with Databricks ML Model Export to generate evaluations, scoring, or...","datacollector/UserGuide/Processors/Delay.html@@@Delay@@@Supported pipeline types: Data Collector Data Collector Edge The Delay processor delays passing a batch to rest of the pipeline by the specified number of milliseconds. Use the Delay processor to...","datacollector/UserGuide/Processors/EncryptDecrypt.html@@@Encrypt and Decrypt Fields@@@When you use the Encrypt and Decrypt Fields processor, you specify the key provider for the stage...","datacollector/UserGuide/Processors/Expression.html@@@Expression Evaluator@@@Supported pipeline types: Data Collector Data Collector Edge The Expression Evaluator performs calculations and writes the results to new or existing fields. You can also use the Expression Evaluator...","datacollector/UserGuide/Processors/FieldFlattener.html@@@Field Flattener@@@Supported pipeline types: Data Collector The Field Flattener processor flattens list and map fields. The processor can flatten the entire record to produce a record with no nested fields. Or it can...","datacollector/UserGuide/Processors/FieldHasher.html@@@Field Hasher@@@Field Hasher provides several methods to hash data. When you hash a field more than once, Field Hasher uses the existing hash when generating the next hash...","datacollector/UserGuide/Processors/FieldMapper.html@@@Field Mapper@@@Supported pipeline types: Data Collector The Field Mapper processor maps an expression to a set of fields to alter field paths, field names, or field values. For example, you might use the Field...","datacollector/UserGuide/Processors/FieldMasker.html@@@Field Masker@@@You can use the following mask types to mask data...","datacollector/UserGuide/Processors/FieldMerger.html@@@Field Merger@@@Supported pipeline types: Data Collector The Field Merger merges one or more fields in a record to a different location in the record. Use only for records with a list or map structure. When you...","datacollector/UserGuide/Processors/FieldOrder.html@@@Field Order@@@When you configure the Field Order processor, you specify the list of fields to order.\n        An incoming record might be missing one of those fields or it might have some extra fields.\n        You configure how the processor handles any missing or extra fields...","datacollector/UserGuide/Processors/FieldRemover.html@@@Field Remover@@@Supported pipeline types: Data Collector Data Collector Edge The Field Remover processor removes fields from records. Use the processor to discard field data that you do not need in the pipeline. When...","datacollector/UserGuide/Processors/FieldRenamer.html@@@Field Renamer@@@You can use regular expressions, or regex, along with the StreamSets expression language to rename sets of fields. You can use regex to define the set of source fields to rename and to define the target field names. You can also use the StreamSets expression language to define target field names...","datacollector/UserGuide/Processors/FieldReplacer.html@@@Field Replacer@@@Use the Field Replacer processor to replace values in a field with null values. You can replace all values with nulls or you can replace a set of values based on a condition...","datacollector/UserGuide/Processors/FieldSplitter.html@@@Field Splitter@@@Supported pipeline types: Data Collector The Field Splitter splits string data based on a regular expression and passes the separated data to new fields. Use the Field Splitter to split complex string...","datacollector/UserGuide/Processors/FieldTypeConverter.html@@@Field Type Converter@@@The following table lists the data types that can be converted to another data type.\n  List, Map, and List-Map data types cannot be converted...","datacollector/UserGuide/Processors/FieldZip.html@@@Field Zip@@@Supported pipeline types: Data Collector The Field Zip processor merges list data from two fields into a single field. You can use the Field Zip processor to merge two List fields or List-Map fields...","datacollector/UserGuide/Processors/GeoIP.html@@@Geo IP@@@Supported pipeline types: Data Collector The Geo IP processor is a lookup processor that can return geolocation and IP intelligence information for a specified IP address. The Geo IP processor uses...","datacollector/UserGuide/Processors/Groovy.html@@@Groovy Evaluator@@@You can choose the processing mode that the Groovy Evaluator uses to process the main script. You can use the same script in each processing mode. However, you should include error handling in the main script before you run in batch mode...","datacollector/UserGuide/Processors/HBaseLookup.html@@@HBase Lookup@@@When you define the lookup key, you specify the row and optionally the column and timestamp to look up in HBase...","datacollector/UserGuide/Processors/HTTPClient.html@@@HTTP Client@@@The HTTP Client processor sends multiple requests at a time. To preserve record order,\n        the processor waits until all requests for the entire batch are completed before processing the next batch...","datacollector/UserGuide/Processors/HTTPRouter.html@@@HTTP Router@@@Supported pipeline types: Data Collector The HTTP Router processor passes records to data streams based on the HTTP method and URL path in the record header attributes. You can use the HTTP Router...","datacollector/UserGuide/Processors/HiveMetadata.html@@@Hive Metadata@@@The Hive Metadata processor queries Hive for information and caches the results. When possible, it uses the cache for record comparison to avoid unnecessary Hive queries...","datacollector/UserGuide/Processors/JDBCLookup.html@@@JDBC Lookup@@@The JDBC Lookup processor can perform lookups of database data from multiple database vendors...","datacollector/UserGuide/Processors/JDBCTee.html@@@JDBC Tee@@@The JDBC Tee processor can write data to multiple database vendors...","datacollector/UserGuide/Processors/JSONGenerator.html@@@JSON Generator@@@Supported pipeline types: Data Collector The JSON Generator serializes data in a field to a JSON-encoded string. You can serialize data from List, Map, or List-Map fields. When you configure the...","datacollector/UserGuide/Processors/JSONParser.html@@@JSON Parser@@@Configure a JSON Parser to parse a JSON object in a String field...","datacollector/UserGuide/Processors/JavaScript.html@@@JavaScript Evaluator@@@You can choose the processing mode that the JavaScript Evaluator uses to process the main script. You can use the same script in each processing mode. However, you should include error handling in the main script before you run in batch mode...","datacollector/UserGuide/Processors/Jython.html@@@Jython Evaluator@@@You can choose the processing mode that the Jython Evaluator uses to process the main script. You can use the same script in each processing mode. However, you should include error handling in the main script before you run in batch mode...","datacollector/UserGuide/Processors/KuduLookup.html@@@Kudu Lookup@@@Supported pipeline types: Data Collector The Kudu Lookup processor performs lookups in a Kudu table and passes the lookup values to fields. Use the Kudu Lookup to enrich records with additional data...","datacollector/UserGuide/Processors/ListPivoter.html@@@Field Pivoter@@@Configure a Field Pivoter to pivot data in a list, map, or list-map field and generate a record for each item in the field...","datacollector/UserGuide/Processors/LogParser.html@@@Log Parser@@@When you use Log Parser to parse log data, you define the format of the log files to be read...","datacollector/UserGuide/Processors/MLeap.html@@@MLeap Evaluator@@@Supported pipeline types: Data Collector The MLeap Evaluator processor uses a machine learning model stored in an MLeap bundle to generate evaluations, scoring, or classifications of data. With the...","datacollector/UserGuide/Processors/MongoDBLookup.html@@@MongoDB Lookup@@@When you configure the MongoDB Lookup processor, you define the document fields to look up in MongoDB. You map these document fields to fields in the record that contain the values to look up...","datacollector/UserGuide/Processors/PMML.html@@@PMML Evaluator@@@Supported pipeline types: Data Collector The PMML Evaluator processor uses a machine learning model stored in the Predictive Model Markup Language (PMML) format to generate predictions or...","datacollector/UserGuide/Processors/PostgreSQLMetadata.html@@@PostgreSQL Metadata@@@Supported pipeline types: Data Collector The PostgreSQL Metadata processor determines the PostgreSQL table where each record should be written, compares the record structure against the table...","datacollector/UserGuide/Processors/Processors_overview.html@@@Processors@@@A processor stage represents a type of data processing that you want to perform. You can use as many processors in a pipeline as you need. You can use different processors based on the execution mode...","datacollector/UserGuide/Processors/Processors_title.html@@@Processors@@@...","datacollector/UserGuide/Processors/RDeduplicator.html@@@Record Deduplicator@@@The Record Deduplicator caches record information for comparison until it reaches a specified number of records. Then, it discards the information in the cache and starts over...","datacollector/UserGuide/Processors/RedisLookup.html@@@Redis Lookup@@@When you define the key to look up in Redis, you specify the data type of the Redis value. The Redis Lookup processor converts the Redis data type to a Data Collector data type...","datacollector/UserGuide/Processors/SQLParser.html@@@SQL Parser@@@Supported pipeline types: Data Collector The SQL Parser parses a SQL query in a string field. When parsing a query, the processor generates fields based on the fields defined in the SQL query and...","datacollector/UserGuide/Processors/SalesforceLookup.html@@@Salesforce Lookup@@@Supported pipeline types: Data Collector The Salesforce Lookup processor performs lookups in a Salesforce object and passes the lookup values to fields. Use the Salesforce Lookup to enrich records...","datacollector/UserGuide/Processors/SchemaGenerator.html@@@Schema Generator@@@Supported pipeline types: Data Collector The Schema Generator processor generates a schema based on the structure of a record and writes the schema into a record header attribute. The Schema Generator...","datacollector/UserGuide/Processors/Spark.html@@@Spark Evaluator@@@Supported pipeline types: Data Collector The Spark Evaluator performs custom processing within a pipeline based on a Spark application that you develop. The Spark Evaluator processor is especially...","datacollector/UserGuide/Processors/StaticLookup.html@@@Static Lookup@@@Configure a Static Lookup processor to perform key-value lookups in memory...","datacollector/UserGuide/Processors/StreamSelector.html@@@Stream Selector@@@The default stream captures records that do not match user-defined conditions. Use the default stream to manage unmatched records...","datacollector/UserGuide/Processors/TensorFlow.html@@@TensorFlow Evaluator@@@The TensorFlow Evaluator processor can evaluate each record or evaluate the entire batch at once...","datacollector/UserGuide/Processors/ValueReplacer.html@@@Value Replacer (Deprecated)@@@The Value Replacer replaces values in fields. However, the Value Replacer is now deprecated and will be removed in a future release. We recommend using the Field Replacer processor. You can use the...","datacollector/UserGuide/Processors/WholeFileTransformer.html@@@Whole File Transformer@@@The Write Avro pipeline reads data from the origin, performs any required processing,\n        and uses a file-based Avro destination such as Local FS to write Avro files to a file system...","datacollector/UserGuide/Processors/XMLFlattener.html@@@XML Flattener@@@Configure an XML Flattener to flatten XML data embedded in a string field...","datacollector/UserGuide/Processors/XMLParser.html@@@XML Parser@@@Configure an XML Parser to parse XML data in a string field...","datacollector/UserGuide/RPC_Pipelines/SDC_RPCpipelines_title.html@@@SDC RPC Pipelines@@@Data Collector Remote Protocol Call pipelines, a.k.a. SDC RPC pipelines, are a set of StreamSets pipelines that pass data from one pipeline to another without writing to an intermediary system...","datacollector/UserGuide/Troubleshooting/Troubleshooting_title.html@@@Troubleshooting@@@Informational and error messages display in different locations based on the type of information: Pipeline configuration issues The Data Collector UI provides guidance and error details as follows...","datacollector/UserGuide/Tutorial/BasicTutorial.html@@@Basic Tutorial@@@The basic tutorial creates a pipeline that reads a file from a directory, processes the data in two branches, and writes all data to a file system. You&apos;ll use data preview to help configure the pipeline, and you&apos;ll create a data alert and run the pipeline...","datacollector/UserGuide/Tutorial/BeforeYouBegin.html@@@Before You Begin@@@Before you start this tutorial, you&apos;ll need to do a few things: Download sample data. You can download sample data from the following location...","datacollector/UserGuide/Tutorial/ExtendedTutorial.html@@@Extended Tutorial@@@The extended tutorial builds on the basic tutorial, using an additional set of stages to perform some data transformations and write to the Trash development destination. We&apos;ll also use data preview to test stage configuration...","datacollector/UserGuide/Tutorial/Overview.html@@@Tutorial Overview@@@This tutorial walks through creating and running a pipeline. You can download sample data so you can perform data preview, run the completed pipeline, and monitor the results...","datacollector/UserGuide/Tutorial/Tutorial-title.html@@@Tutorial@@@...","datacollector/UserGuide/Upgrade/CMUpgrade.html@@@Upgrade an Installation with Cloudera Manager@@@In Data Collector, stop all running pipelines...","datacollector/UserGuide/Upgrade/PostUpgrade.html@@@Post Upgrade Tasks@@@In some situations, you must complete tasks within Data Collector or your Control Hub on-premises installation after you upgrade...","datacollector/UserGuide/Upgrade/PreUpgrade.html@@@Pre Upgrade Tasks@@@In some situations, you must complete tasks before you upgrade...","datacollector/UserGuide/Upgrade/RPM.html@@@Upgrade an Installation from the RPM Package@@@When you upgrade an installation from the RPM package, the new version uses the default configuration, data, log, and resource directories. If the previous version used the default directories, the new version has access to the files created in the previous version...","datacollector/UserGuide/Upgrade/Tarball.html@@@Upgrade an Installation from the Tarball@@@Stop all pipelines and then shut down the previous version of Data Collector...","datacollector/UserGuide/Upgrade/Upgrade-ExternalSystems.html@@@Working with Upgraded External Systems@@@When an external system is upgraded to a new version, you can continue to use existing Data Collector pipelines that connected to the previous version of the external system. You simply configure the pipelines to work with the upgraded system...","datacollector/UserGuide/Upgrade/Upgrade.html@@@Upgrade@@@You can upgrade a previous version of Data Collector to a new version. You can upgrade an installation from the tarball, an installation from the RPM package, or an installation with Cloudera Manager...","datacollector/UserGuide/Upgrade/UpgradeTroubleshooting.html@@@Troubleshooting an Upgrade@@@Use the following tips for help with upgrades: After upgrading a Data Collector that is registered with StreamSets Control Hub , the Data Collector fails to start with the following error about a...","datacollector/UserGuide/Upgrade/Upgrade_title.html@@@Upgrade@@@...","datacollector/UserGuide/WhatsNew/WhatsNew_Title.html@@@What&apos;s New@@@Data Collector version 3.8.0 includes the following new features and enhancements: Memory Monitoring With version 3.8.0, Data Collector memory monitoring has been removed. Memory monitoring was..."];
  });