Key Default Type Description
external-resource.<resource_name>.yarn.config-key
(none) String If configured, Flink will add this key to the resource profile of container request to Yarn. The value will be set to the value of external-resource.<resource_name>.amount.
flink.hadoop.<key>
(none) String A general option to probe Hadoop configuration through prefix 'flink.hadoop.'. Flink will remove the prefix to get <key> (from core-default.xml and hdfs-default.xml) then set the <key> and value to Hadoop configuration. For example, flink.hadoop.dfs.replication=5 in Flink configuration and convert to dfs.replication=5 in Hadoop configuration.
flink.yarn.<key>
(none) String A general option to probe Yarn configuration through prefix 'flink.yarn.'. Flink will remove the prefix 'flink.' to get yarn.<key> (from yarn-default.xml) then set the yarn.<key> and value to Yarn configuration. For example, flink.yarn.resourcemanager.container.liveness-monitor.interval-ms=300000 in Flink configuration and convert to yarn.resourcemanager.container.liveness-monitor.interval-ms=300000 in Yarn configuration.
yarn.application-attempt-failures-validity-interval
10000 Long Time window in milliseconds which defines the number of application attempt failures when restarting the AM. Failures which fall outside of this window are not being considered. Set this value to -1 in order to count globally. See here for more information.
yarn.application-attempts
(none) String Number of ApplicationMaster restarts. By default, the value will be set to 1. If high availability is enabled, then the default value will be 2. The restart number is also limited by YARN (configured via yarn.resourcemanager.am.max-attempts). Note that the entire Flink cluster will restart and the YARN Client will lose the connection.
yarn.application-master.port
"0" String With this configuration option, users can specify a port, a range of ports or a list of ports for the Application Master (and JobManager) RPC port. By default we recommend using the default value (0) to let the operating system choose an appropriate port. In particular when multiple AMs are running on the same physical host, fixed port assignments prevent the AM from starting. For example when running Flink on YARN on an environment with a restrictive firewall, this option allows specifying a range of allowed ports.
yarn.application.id
(none) String The YARN application id of the running yarn cluster. This is the YARN cluster where the pipeline is going to be executed.
yarn.application.name
(none) String A custom name for your YARN application.
yarn.application.node-label
(none) String Specify YARN node label for the YARN application.
yarn.application.priority
-1 Integer A non-negative integer indicating the priority for submitting a Flink YARN application. It will only take effect if YARN priority scheduling setting is enabled. Larger integer corresponds with higher priority. If priority is negative or set to '-1'(default), Flink will unset yarn priority setting and use cluster default priority. Please refer to YARN's official documentation for specific settings required to enable priority scheduling for the targeted YARN version.
yarn.application.queue
(none) String The YARN queue on which to put the current pipeline.
yarn.application.type
(none) String A custom type for your YARN application..
yarn.appmaster.vcores
1 Integer The number of virtual cores (vcores) used by YARN application master.
yarn.classpath.include-user-jar
ORDER

Enum

Defines whether user-jars are included in the system class path as well as their positioning in the path.

Possible values:
  • "DISABLED": Exclude user jars from the system class path
  • "FIRST": Position at the beginning
  • "LAST": Position at the end
  • "ORDER": Position based on the name of the jar
yarn.container-start-command-template
"%java% %jvmmem% %jvmopts% %logging% %class% %args% %redirects%" String This configuration parameter allows users to pass custom settings (such as JVM paths, arguments etc.) to start the YARN. The following placeholders will be replaced:
  • %java%: Path to the Java executable
  • %jvmmem%: JVM memory limits and tweaks
  • %jvmopts%: Options for the Java VM
  • %logging%: Logging-related configuration settings
  • %class%: Main class to execute
  • %args%: Arguments for the main class
  • %redirects%: Output redirects
yarn.containers.vcores
-1 Integer The number of virtual cores (vcores) per YARN container. By default, the number of vcores is set to the number of slots per TaskManager, if set, or to 1, otherwise. In order for this parameter to be used your cluster must have CPU scheduling enabled. You can do this by setting the org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.
yarn.file-replication
-1 Integer Number of file replication of each local resource file. If it is not configured, Flink will use the default replication value in hadoop configuration.
yarn.flink-dist-jar
(none) String The location of the Flink dist jar.
yarn.heartbeat.container-request-interval
500 Integer Time between heartbeats with the ResourceManager in milliseconds if Flink requests containers:
  • The lower this value is, the faster Flink will get notified about container allocations since requests and allocations are transmitted via heartbeats.
  • The lower this value is, the more excessive containers might get allocated which will eventually be released but put pressure on Yarn.
If you observe too many container allocations on the ResourceManager, then it is recommended to increase this value. See this link for more information.
yarn.heartbeat.interval
5 Integer Time between heartbeats with the ResourceManager in seconds.
yarn.modify.acls
(none) String Users and groups to give MODIFY access. The ACLs are of for comma-separated-users&lt;space&gt;comma-separated-groups. Wildcard ACL is also supported. The only valid wildcard ACL is *, which grants permission to all users and groups.
yarn.properties-file.location
(none) String When a Flink job is submitted to YARN, the JobManager’s host and the number of available processing slots is written into a properties file, so that the Flink client is able to pick those details up. This configuration parameter allows changing the default location of that file (for example for environments sharing a Flink installation between users).
yarn.provided.lib.dirs
(none) List<String> A semicolon-separated list of provided lib directories. They should be pre-uploaded and world-readable. Flink will use them to exclude the local Flink jars(e.g. flink-dist, lib/, plugins/)uploading to accelerate the job submission process. Also YARN will cache them on the nodes so that they doesn't need to be downloaded every time for each application. An example could be hdfs://$namenode_address/path/of/flink/lib
yarn.provided.usrlib.dir
(none) String The provided usrlib directory in remote. It should be pre-uploaded and world-readable. Flink will use it to exclude the local usrlib directory(i.e. usrlib/ under the parent directory of FLINK_LIB_DIR). Unlike yarn.provided.lib.dirs, YARN will not cache it on the nodes as it is for each application. An example could be hdfs://$namenode_address/path/of/flink/usrlib
yarn.security.kerberos.localized-keytab-path
"krb5.keytab" String Local (on NodeManager) path where kerberos keytab file will be localized to. If yarn.security.kerberos.ship-local-keytab set to true, Flink will ship the keytab file as a YARN local resource. In this case, the path is relative to the local resource directory. If set to false, Flink will try to directly locate the keytab from the path itself.
yarn.security.kerberos.ship-local-keytab
true Boolean When this is true Flink will ship the keytab file configured via security.kerberos.login.keytab as a localized YARN resource.
yarn.ship-archives
(none) List<String> A semicolon-separated list of archives to be shipped to the YARN cluster. These archives can come from the local path of flink client or HDFS. They will be un-packed when localizing and they can be any of the following types: ".tar.gz", ".tar", ".tgz", ".dst", ".jar", ".zip". For example, "/path/to/local/archive.jar;hdfs://$namenode_address/path/to/archive.jar"
yarn.ship-files
(none) List<String> A semicolon-separated list of files and/or directories to be shipped to the YARN cluster. These files/directories can come from the local path of flink client or HDFS. For example, "/path/to/local/file;/path/to/local/directory;hdfs://$namenode_address/path/of/file;hdfs://$namenode_address/path/of/directory"
yarn.staging-directory
(none) String Staging directory used to store YARN files while submitting applications. Per default, it uses the home directory of the configured file system.
yarn.tags
(none) String A comma-separated list of tags to apply to the Flink YARN application.
yarn.taskmanager.node-label
(none) String Specify YARN node label for the Flink TaskManagers, it will override the yarn.application.node-label for TaskManagers if both are set.
yarn.view.acls
(none) String Users and groups to give VIEW access. The ACLs are of for comma-separated-users&lt;space&gt;comma-separated-groups. Wildcard ACL is also supported. The only valid wildcard ACL is *, which grants permission to all users and groups.