The EMRFS CLI is installed by default on all cluster master nodes created using Amazon EMR release version 3.2.1 or later. You can use the EMRFS CLI to manage the metadata for consistent view.
The emrfs command is only supported with VT100 terminal emulation. However, it may work with other terminal emulator modes.
The emrfs top-level command supports the following structure.
Specify [options], with or without [arguments] as described in the following table. For [options] specific to sub-commands (describe-metadata, set-metadata-capacity, etc.), see each sub-command below.
The AWS access key you use to write objects to Amazon S3 and to create or access a metadata store in DynamoDB. By default, AWS_ACCESS_KEY_ID is set to the access key used to create the cluster.
The AWS secret key associated with the access key you use to write objects to Amazon S3 and to create or access a metadata store in DynamoDB. By default, AWS_SECRET_ACCESS_KEY is set to the secret key associated with the access key used to create the cluster.
Displays the help message for the emrfs command with a usage statement.
METADATA_NAME is the name of the DynamoDB metadata table. If the METADATA_NAME argument is not supplied, the default value is EmrFSMetadata.
The following example describes the default metadata table.
The requested read throughput capacity for the metadata table. If the READ_CAPACITY argument is not supplied, the default value is 400.
The requested write throughput capacity for the metadata table. If the WRITE_CAPACITY argument is not supplied, the default value is 100.
The following example sets the read throughput capacity to 600 and the write capacity to 150 for a metadata table named EmrMetadataAlt.
The following example deletes the default metadata table.
The following example creates a metadata table named EmrFSMetadataAlt.
The emrfs list-metadata-stores sub-command has no [options].
The following example lists your metadata tables.
The path to the Amazon S3 bucket to compare with the metadata table. Buckets sync recursively.
The following example compares the default metadata table to an Amazon S3 bucket.
The path to the Amazon S3 bucket you are tracking for consistent view. Buckets sync recursively.
The expiration time (interpreted using the time unit argument). All metadata entries older than the TIME argument are deleted for the specified bucket.
The measure used to interpret the time argument (nanoseconds, microseconds, milliseconds, seconds, minutes, hours, or days). If no argument is specified, the default value is days.
The requested amount of available read throughput used for the delete operation. If the READ_CONSUMPTION argument is not specified, the default value is 400.
The requested amount of available write throughput used for the delete operation. If the WRITE_CONSUMPTION argument is not specified, the default value is 100.
The following example removes all objects in an Amazon S3 bucket from the tracking metadata for consistent view.
The following example imports all objects in an Amazon S3 bucket with the tracking metadata for consistent view. All unknown keys are ignored.
The following example imports all objects in an Amazon S3 bucket with the tracking metadata for consistent view. All unknown keys are deleted.
QUEUE_NAME is the name of the Amazon SQS queue configured in emrfs-site.xml. The default value is EMRFS-Inconsistency-<jobFlowId>.
OUTPUT_FILE is the path to the output file on the master node's local file system. Messages read from the queue are written to this file.
The following example shows how to use the emrfs utility on the master node by leveraging the AWS CLI or API and the script-runner.jar to run the emrfs command as a step. The example uses the AWS SDK for Python (Boto) to add a step to a cluster which adds objects in an Amazon S3 bucket to the default EMRFS metadata table. Replace j-2AL4XXXXXX5T9 with the ID of the cluster.
You can use the stepId value returned to check the logs for the result of the operation.