自定义服务配置 本章节主要介绍自定义服务配置的使用方法。 操作场景 HDFS、YARN、Hive等服务含有大量配置项,为便于服务使用,您可使用自定义服务配置功能,在创建集群或新增服务前修改或添加配置项。 前提条件 当前支持用户对HDFS、Hive、Spark、HBase、YARN、Kyuubi的部分文件进行修改与添加配置项。其他组件服务暂不支持使用自定义服务配置功能。 操作步骤 1、支持用户在订购环节对所选服务进行配置修改与添加。 1)在翼MR订购页软件配置环节选择所需组件服务。 2)在软件配置环节,开启自定义服务配置开关。 3)在输入框中添加JSON 格式的配置信息。 2、支持用户在新增集群服务环节对所选服务的配置进行修改与添加。 1)在翼MR控制台集群服务管理页面选择需要新增部署的组件服务。 2)在弹窗中开启自定义服务配置开关。 3)在输入框中添加JSON 格式的配置信息。 配置说明 在创建集群或新增集群服务过程中,可通过自定义服务配置功能,添加JSON格式的配置文件,覆盖或添加集群服务的默认参数。JSON文件的内容示例如下: plaintext [ { "applicationName":"YARN", "configFileName":"mapredsite.xml", "configItemKey":"mapreduce.task.timeout", "configItemValue":"200000" }, { "applicationName":"HDFS", "configFileName":"hdfssite.xml", "configItemKey":"dfs.replication", "configItemValue":"2" } ] 参数含义说明: 参数名称 说明 applicationName 服务的名称。 configFileName 配置文件的名称。 configItemKey 配置项的名称。 configItemValue 该配置需要设置的具体内容。 针对不同文件自定义服务配置的修改方式有所不同 1、全文修改 部分配置文件只能进行全文覆盖。用户需要使用CONFCONTENT作为configItemKey,在configItemValue中输入转义后的配置文件内容,输入的配置文件内容会在部署后自动覆盖到服务对应的配置上。文件内容的字符串需要用户对常见字符进行转义,如双引号、单引号、换行符、反斜杠等。需要通过全文覆盖方式进行配置修改的文件如下: 服务名称 配置文件 HBase hbaseenv.sh HBase log4j.properties HDFS hadoopenv.sh YARN mapredenv.sh YARN yarnenv.sh Hive hiveenv.sh Hive hivelog4j2.properties Kyuubi kyuubienv.sh 2、按配置项修改 部分配置文件支持按配置项进行修改,支持修改的配置项如下: 说明 1)Kerberos的krb5.conf文件,只支持使用kerberosRealm作为configItemKey来配realm,暂不支持配置其它配置项。 2)Spark的sparkenv.sh文件中,customsparkenv配置项需要对参照上文全文修改的方式,对文件内容进行转义。 服务名称 配置文件 配置项 HBase hbasesite.xml hbase.cluster.distributed HBase hbasesite.xml hbase.coprocessor.abortonerror HBase hbasesite.xml hbase.coprocessor.master.classes HBase hbasesite.xml hbase.coprocessor.region.classes HBase hbasesite.xml hbase.master.kerberos.principal HBase hbasesite.xml hbase.master.keytab.file HBase hbasesite.xml hbase.master.loadbalancer.class HBase hbasesite.xml hbase.quota.enabled HBase hbasesite.xml hbase.quota.refresh.period HBase hbasesite.xml hbase.regionserver.kerberos.principal HBase hbasesite.xml hbase.regionserver.keytab.file HBase hbasesite.xml hbase.rest.authentication.kerberos.keytab HBase hbasesite.xml hbase.rest.authentication.kerberos.principal HBase hbasesite.xml hbase.rest.kerberos.principal HBase hbasesite.xml hbase.rootdir HBase hbasesite.xml hbase.security.authentication HBase hbasesite.xml hbase.security.authorization HBase hbasesite.xml hbase.superuser HBase hbasesite.xml hbase.thrift.kerberos.principal HBase hbasesite.xml hbase.thrift.keytab.file HBase hbasesite.xml hbase.tmp.dir HBase hbasesite.xml hbase.unsafe.stream.capability.enforce HBase hbasesite.xml hbase.wal.provider HBase hbasesite.xml hbase.zookeeper.property.clientPort HBase hbasesite.xml hbase.zookeeper.quorum HBase hbasesite.xml zookeeper.znode.parent HDFS coresite.xml dfs.datanode.cacheddfsused.check.interval.ms HDFS coresite.xml fs.AbstractFileSystem.jfs.impl HDFS coresite.xml fs.defaultFS HDFS coresite.xml fs.du.interval HDFS coresite.xml fs.getspaceused.jitterMillis HDFS coresite.xml fs.jfs.impl HDFS coresite.xml fs.permissions.umaskmode HDFS coresite.xml fs.trash.checkpoint.interval HDFS coresite.xml fs.trash.interval HDFS coresite.xml ha.failovercontroller.newactive.rpctimeout.m HDFS coresite.xml ha.failovercontroller.newactive.rpctimeout.ms HDFS coresite.xml ha.healthmonitor.rpctimeout.ms HDFS coresite.xml ha.zookeeper.quorum HDFS coresite.xml ha.zookeeper.sessiontimeout.ms HDFS coresite.xml hadoop.caller.context.enabled HDFS coresite.xml hadoop.kerberos.kinit.command HDFS coresite.xml hadoop.proxyuser.hdfs.groups HDFS coresite.xml hadoop.proxyuser.hdfs.hosts HDFS coresite.xml hadoop.proxyuser.hive.groups HDFS coresite.xml hadoop.proxyuser.hive.hosts HDFS coresite.xml hadoop.proxyuser.HTTP.groups HDFS coresite.xml hadoop.proxyuser.HTTP.hosts HDFS coresite.xml hadoop.proxyuser.httpfs.groups HDFS coresite.xml hadoop.proxyuser.httpfs.hosts HDFS coresite.xml hadoop.proxyuser.yarn.groups HDFS coresite.xml hadoop.proxyuser.yarn.hosts HDFS coresite.xml hadoop.rpc.protection HDFS coresite.xml hadoop.security.authtolocal HDFS coresite.xml hadoop.security.authentication HDFS coresite.xml hadoop.security.authorization HDFS coresite.xml hadoop.security.group.mapping HDFS coresite.xml hadoop.security.group.mapping.ldap.base HDFS coresite.xml hadoop.security.group.mapping.ldap.bind.password.file HDFS coresite.xml hadoop.security.group.mapping.ldap.bind.user HDFS coresite.xml hadoop.security.group.mapping.ldap.num.attempts HDFS coresite.xml hadoop.security.group.mapping.ldap.num.attempts.before.failover HDFS coresite.xml hadoop.security.group.mapping.ldap.posix.attr.uid.name HDFS coresite.xml hadoop.security.group.mapping.ldap.search.attr.group.name HDFS coresite.xml hadoop.security.group.mapping.ldap.search.attr.member HDFS coresite.xml hadoop.security.group.mapping.ldap.search.filter.group HDFS coresite.xml hadoop.security.group.mapping.ldap.search.filter.user HDFS coresite.xml hadoop.security.group.mapping.provider.ldap4users HDFS coresite.xml hadoop.security.group.mapping.provider.ldap4users.ldap.url HDFS coresite.xml hadoop.security.group.mapping.provider.shell4services HDFS coresite.xml hadoop.security.group.mapping.providers HDFS coresite.xml hadoop.security.group.mapping.providers.combined HDFS coresite.xml httpfs.proxyuser.mapred.groups HDFS coresite.xml httpfs.proxyuser.mapred.hosts HDFS coresite.xml io.compression.codec.lzo.class HDFS coresite.xml io.file.buffer.size HDFS coresite.xml ipc.client.connection.maxidletime HDFS coresite.xml ipc.server.listen.queue.size HDFS coresite.xml ipc.server.log.slow.rpc HDFS coresite.xml jeekefs.cachedir HDFS coresite.xml jeekefs.cachesize HDFS coresite.xml jeekefs.discovernodesurl HDFS coresite.xml jeekefs.meta HDFS coresite.xml jeekefs.serverprincipal HDFS coresite.xml topology.script.file.name HDFS hdfssite.xml dfs.block.access.token.enable HDFS hdfssite.xml dfs.blockreport.incremental.intervalMsec HDFS hdfssite.xml dfs.blockreport.initialDelay HDFS hdfssite.xml dfs.blockreport.intervalMsec HDFS hdfssite.xml dfs.blockreport.split.threshold HDFS hdfssite.xml dfs.blocksize HDFS hdfssite.xml dfs.client.failover.proxy.provider.ctyunns HDFS hdfssite.xml dfs.client.read.shortcircuit HDFS hdfssite.xml dfs.client.sockettimeout HDFS hdfssite.xml dfs.cluster.administrators HDFS hdfssite.xml dfs.datanode.address HDFS hdfssite.xml dfs.datanode.cacheddfsused.check.interval.ms HDFS hdfssite.xml dfs.datanode.data.dir HDFS hdfssite.xml dfs.datanode.data.dir.perm HDFS hdfssite.xml dfs.datanode.directoryscan.threads HDFS hdfssite.xml dfs.datanode.du.reserved.calculator HDFS hdfssite.xml dfs.datanode.du.reserved.pct HDFS hdfssite.xml dfs.datanode.failed.volumes.tolerated HDFS hdfssite.xml dfs.datanode.fileio.profiling.sampling.percentage HDFS hdfssite.xml dfs.datanode.handler.count HDFS hdfssite.xml dfs.datanode.http.address HDFS hdfssite.xml dfs.datanode.kerberos.principal HDFS hdfssite.xml dfs.datanode.keytab.file HDFS hdfssite.xml dfs.datanode.max.transfer.threads HDFS hdfssite.xml dfs.datanode.max.xcievers HDFS hdfssite.xml dfs.datanode.peer.stats.enabled HDFS hdfssite.xml dfs.domain.socket.path HDFS hdfssite.xml dfs.encrypt.data.transfer.cipher.suites HDFS hdfssite.xml dfs.ha.automaticfailover.enabled HDFS hdfssite.xml dfs.ha.fencing.methods HDFS hdfssite.xml dfs.ha.namenodes.ctyunns HDFS hdfssite.xml dfs.hosts.exclude HDFS hdfssite.xml dfs.image.transfer.bandwidthPerSec HDFS hdfssite.xml dfs.internal.nameservices HDFS hdfssite.xml dfs.journalnode.edits.dir.ctyunns HDFS hdfssite.xml dfs.journalnode.httpaddress HDFS hdfssite.xml dfs.journalnode.kerberos.internal.spnego.principal HDFS hdfssite.xml dfs.journalnode.kerberos.principal HDFS hdfssite.xml dfs.journalnode.keytab.file HDFS hdfssite.xml dfs.journalnode.rpcaddress HDFS hdfssite.xml dfs.namenode.accesstime.precision HDFS hdfssite.xml dfs.namenode.acls.enabled HDFS hdfssite.xml dfs.namenode.audit.log.async HDFS hdfssite.xml dfs.namenode.avoid.read.stale.datanode HDFS hdfssite.xml dfs.namenode.avoid.write.stale.datanode HDFS hdfssite.xml dfs.namenode.block.deletion.increment HDFS hdfssite.xml dfs.namenode.checkpoint.dir HDFS hdfssite.xml dfs.namenode.checkpoint.edits.dir HDFS hdfssite.xml dfs.namenode.checkpoint.period HDFS hdfssite.xml dfs.namenode.checkpoint.txns HDFS hdfssite.xml dfs.namenode.deletefiles.limit HDFS hdfssite.xml dfs.namenode.edit.log.autoroll.multiplier.threshold HDFS hdfssite.xml dfs.namenode.fslimits.maxdirectoryitems HDFS hdfssite.xml dfs.namenode.fslock.fair HDFS hdfssite.xml dfs.namenode.handler.count HDFS hdfssite.xml dfs.namenode.httpaddress.ctyunns.nn1 HDFS hdfssite.xml dfs.namenode.httpaddress.ctyunns.nn2 HDFS hdfssite.xml dfs.namenode.kerberos.internal.spnego.principal HDFS hdfssite.xml dfs.namenode.kerberos.principal HDFS hdfssite.xml dfs.namenode.keytab.file HDFS hdfssite.xml dfs.namenode.lock.detailedmetrics.enabled HDFS hdfssite.xml dfs.namenode.name.dir HDFS hdfssite.xml dfs.namenode.name.dir.restore HDFS hdfssite.xml dfs.namenode.quota.initthreads HDFS hdfssite.xml dfs.namenode.rpcaddress.ctyunns.nn1 HDFS hdfssite.xml dfs.namenode.rpcaddress.ctyunns.nn2 HDFS hdfssite.xml dfs.namenode.safemode.thresholdpct HDFS hdfssite.xml dfs.namenode.service.handler.count HDFS hdfssite.xml dfs.namenode.servicerpcaddress.ctyunns.nn1 HDFS hdfssite.xml dfs.namenode.servicerpcaddress.ctyunns.nn2 HDFS hdfssite.xml dfs.namenode.shared.edits.dir.ctyunns HDFS hdfssite.xml dfs.namenode.stale.datanode.interval HDFS hdfssite.xml dfs.namenode.startup.delay.block.deletion.sec HDFS hdfssite.xml dfs.namenode.support.allow.format HDFS hdfssite.xml dfs.namenode.write.stale.datanode.ratio HDFS hdfssite.xml dfs.nameservices HDFS hdfssite.xml dfs.permissions.superusergroup HDFS hdfssite.xml dfs.qjournal.selectinputstreams.timeout.ms HDFS hdfssite.xml dfs.qjournal.startsegment.timeout.ms HDFS hdfssite.xml dfs.qjournal.writetxns.timeout.ms HDFS hdfssite.xml dfs.replication HDFS hdfssite.xml dfs.replication.max HDFS hdfssite.xml dfs.web.authentication.kerberos.keytab HDFS hdfssite.xml dfs.web.authentication.kerberos.principal HDFS hdfssite.xml dfs.webhdfs.enabled HDFS hdfssite.xml hadoop.caller.context.enabled HDFS hdfssite.xml rpc.metrics.percentiles.intervals HDFS hdfssite.xml rpc.metrics.quantile.enable Hive hivesite.xml hive.auto.convert.join Hive hivesite.xml hive.auto.convert.sortmerge.join Hive hivesite.xml hive.auto.convert.sortmerge.join.to.mapjoin Hive hivesite.xml hive.compactor.initiator.on Hive hivesite.xml hive.default.fileformat Hive hivesite.xml hive.default.fileformat.managed Hive hivesite.xml hive.exec.compress.output Hive hivesite.xml hive.exec.dynamic.partition Hive hivesite.xml hive.exec.stagingdir Hive hivesite.xml hive.execution.engine Hive hivesite.xml hive.hook.proto.basedirectory Hive hivesite.xml hive.insert.into.multilevel.dirs Hive hivesite.xml hive.limit.optimize.enable Hive hivesite.xml hive.mapred.reduce.tasks.speculative.execution Hive hivesite.xml hive.merge.mapredfiles Hive hivesite.xml hive.metastore.authorization.storage.checks Hive hivesite.xml hive.metastore.warehouse.dir Hive hivesite.xml hive.metastore.warehouse.external.dir Hive hivesite.xml hive.optimize.bucketmapjoin Hive hivesite.xml hive.optimize.dynamic.partition.hashjoin Hive hivesite.xml hive.optimize.index.filter Hive hivesite.xml hive.optimize.metadataonly Hive hivesite.xml hive.optimize.remove.identity.project Hive hivesite.xml hive.server2.proxy.user Hive hivesite.xml hive.stats.fetch.column.stats Hive hivesite.xml hive.txn.strict.locking.mode Hive hivesite.xml hive.update.last.access.time.interval Hive hivesite.xml hive.user.install.directory Hive hivesite.xml hive.vectorized.execution.mapjoin.minmax.enabled Hive hivesite.xml hive.vectorized.execution.mapjoin.native.fast.hashtable.enabled Hive hivesite.xml hive.vectorized.groupby.checkinterval Hive hivesite.xml metastore.expression.proxy Kyuubi kyuubidefaults.conf kyuubi.backend.server.event.json.log.path Kyuubi kyuubidefaults.conf kyuubi.backend.server.event.loggers Kyuubi kyuubidefaults.conf kyuubi.delegation.token.renew.interval Kyuubi kyuubidefaults.conf kyuubi.ha.namespace Kyuubi kyuubidefaults.conf kyuubi.metrics.reporters Kyuubi kyuubidefaults.conf kyuubi.operation.getTables.ignoreTableProperties Kyuubi kyuubidefaults.conf kyuubi.session.engine.idle.timeout Kyuubi kyuubidefaults.conf kyuubi.session.idle.timeout Kyuubi kyuubidefaults.conf spark.master Kyuubi kyuubidefaults.conf spark.submit.deployMode Kyuubi kyuubidefaults.conf spark.yarn.queue Spark sparkdefaults.conf spark.driver.cores Spark sparkdefaults.conf spark.driver.extraJavaOptions Spark sparkdefaults.conf spark.driver.extraLibraryPath Spark sparkdefaults.conf spark.driver.maxResultSize Spark sparkdefaults.conf spark.driver.memory Spark sparkdefaults.conf spark.dynamicAllocation.enabled Spark sparkdefaults.conf spark.dynamicAllocation.initialExecutors Spark sparkdefaults.conf spark.dynamicAllocation.maxExecutors Spark sparkdefaults.conf spark.dynamicAllocation.minExecutors Spark sparkdefaults.conf spark.eventLog.dir Spark sparkdefaults.conf spark.executor.cores Spark sparkdefaults.conf spark.executor.extraJavaOptions Spark sparkdefaults.conf spark.executor.extraLibraryPath Spark sparkdefaults.conf spark.executor.heartbeatInterval Spark sparkdefaults.conf spark.executor.memory Spark sparkdefaults.conf spark.executorEnv.JAVAHOME Spark sparkdefaults.conf spark.files.openCostInBytes Spark sparkdefaults.conf spark.hadoop.mapreduce.output.fileoutputformat.compress Spark sparkdefaults.conf spark.hadoop.mapreduce.output.fileoutputformat.compress.codec Spark sparkdefaults.conf spark.hadoop.yarn.timelineservice.enabled Spark sparkdefaults.conf spark.history.fs.cleaner.enabled Spark sparkdefaults.conf spark.history.fs.cleaner.interval Spark sparkdefaults.conf spark.history.fs.cleaner.maxAge Spark sparkdefaults.conf spark.history.fs.logDirectory Spark sparkdefaults.conf spark.history.kerberos.enabled Spark sparkdefaults.conf spark.history.kerberos.keytab Spark sparkdefaults.conf spark.history.store.maxDiskUsage Spark sparkdefaults.conf spark.history.ui.maxApplications Spark sparkdefaults.conf spark.history.ui.port Spark sparkdefaults.conf spark.io.compression.lz4.blockSize Spark sparkdefaults.conf spark.kryo.unsafe Spark sparkdefaults.conf spark.kryoserializer.buffer.max Spark sparkdefaults.conf spark.locality.wait Spark sparkdefaults.conf spark.master Spark sparkdefaults.conf spark.memory.offHeap.enabled Spark sparkdefaults.conf spark.memory.offHeap.size Spark sparkdefaults.conf spark.network.timeout Spark sparkdefaults.conf spark.port.maxRetries Spark sparkdefaults.conf spark.rdd.parallelListingThreshold Spark sparkdefaults.conf spark.reducer.maxSizeInFlight Spark sparkdefaults.conf spark.resultGetter.threads Spark sparkdefaults.conf spark.rpc.io.backLog Spark sparkdefaults.conf spark.scheduler.maxReqisteredResourcesWaitingTime Spark sparkdefaults.conf spark.shuffle.accurateBlockThreshold Spark sparkdefaults.conf spark.shuffle.file.buffer Spark sparkdefaults.conf spark.shuffle.io.connectionTimeout Spark sparkdefaults.conf spark.shuffle.manager Spark sparkdefaults.conf spark.shuffle.mapOutput.dispatcher.numThreads Spark sparkdefaults.conf spark.shuffle.memoryFraction Spark sparkdefaults.conf spark.shuffle.push.enabled Spark sparkdefaults.conf spark.shuffle.push.maxBlockSizeToPush Spark sparkdefaults.conf spark.shuffle.push.merge.finalizeThreads Spark sparkdefaults.conf spark.shuffle.push.mergersMinStaticThreshold Spark sparkdefaults.conf spark.shuffle.readHostLocalDisk Spark sparkdefaults.conf spark.shuffle.service.enabled Spark sparkdefaults.conf spark.shuffle.unsafe.file.output.buffer Spark sparkdefaults.conf spark.speculation Spark sparkdefaults.conf spark.speculation.interval Spark sparkdefaults.conf spark.speculation.minTaskRuntime Spark sparkdefaults.conf spark.speculation.multiplier Spark sparkdefaults.conf spark.speculation.quantile Spark sparkdefaults.conf spark.sql.adaptive.coalescePartitions.initialPartitionNum Spark sparkdefaults.conf spark.sql.adaptive.coalescePartitions.minPartitionNum Spark sparkdefaults.conf spark.sql.adaptive.enabled Spark sparkdefaults.conf spark.sql.adaptive.forceApply Spark sparkdefaults.conf spark.sql.adaptive.forceOptimizeSkewedJoin Spark sparkdefaults.conf spark.sql.adaptive.shuffle.targetPostShuffleInputSize Spark sparkdefaults.conf spark.sql.autoBroadcastJoinThreshold Spark sparkdefaults.conf spark.sql.catalog.sparkcatalog Spark sparkdefaults.conf spark.sql.catalog.sparkcatalog.type Spark sparkdefaults.conf spark.sql.cbo.joinReorder.enabled Spark sparkdefaults.conf spark.sql.extensions Spark sparkdefaults.conf spark.sql.files.maxPartitionBytes Spark sparkdefaults.conf spark.sql.files.openCostInBytes Spark sparkdefaults.conf spark.sql.finalStage.adaptive.advisoryPartitionSizeInBytes Spark sparkdefaults.conf spark.sql.finalStage.adaptive.coalescePartitions.minPartitionNum Spark sparkdefaults.conf spark.sql.finalStage.adaptive.skewJoin.skewedPartitionFactor Spark sparkdefaults.conf spark.sql.finalStage.adaptive.skewJoin.skewedPartitionThresholdInBytes Spark sparkdefaults.conf spark.sql.hive.convertMetastoreOrc Spark sparkdefaults.conf spark.sql.hive.dropPartitionByName.enabled Spark sparkdefaults.conf spark.sql.inMemoryColumnarStorage.batchSize Spark sparkdefaults.conf spark.sql.join.preferSortMergeJoin Spark sparkdefaults.conf spark.sql.legacy.charVarcharAsString Spark sparkdefaults.conf spark.sql.legacy.timeParserPolicy Spark sparkdefaults.conf spark.sql.optimizer.finalStageConfigIsolation.enabled Spark sparkdefaults.conf spark.sql.optimizer.inferRebalanceAndSortOrders.enabled Spark sparkdefaults.conf spark.sql.optimizer.insertRepartitionBeforeWriteIfNoShuffle.enabled Spark sparkdefaults.conf spark.sql.optimizer.inSetConversionThreshold Spark sparkdefaults.conf spark.sql.optimizer.runtime.bloomFilter.creationSideThreshold Spark sparkdefaults.conf spark.sql.optimizer.runtime.bloomFilter.enabled Spark sparkdefaults.conf spark.sql.optimizer.runtimeFilter.number.threshold Spark sparkdefaults.conf spark.sql.optimizer.runtimeFilter.semiJoinReduction.enabled Spark sparkdefaults.conf spark.sql.orc.aggregatePushdown Spark sparkdefaults.conf spark.sql.orc.columnarReaderBatchSize Spark sparkdefaults.conf spark.sql.orc.enableNestedColumnVectorizedReader Spark sparkdefaults.conf spark.sql.parquet.aggregatePushdown Spark sparkdefaults.conf spark.sql.parquet.columnarReaderBatchSize Spark sparkdefaults.conf spark.sql.parquet.enableNestedColumnVectorizedReader Spark sparkdefaults.conf spark.sql.parquet.pushdown.inFilterThreshold Spark sparkdefaults.conf spark.sql.query.table.file.max.count Spark sparkdefaults.conf spark.sql.query.table.file.max.length Spark sparkdefaults.conf spark.sql.query.table.partition.max.count Spark sparkdefaults.conf spark.sql.sessionWindow.buffer.in.memory.threshold Spark sparkdefaults.conf spark.sql.shuffle.partitions Spark sparkdefaults.conf spark.sql.sources.parallelPartitionDiscovery.parallelism Spark sparkdefaults.conf spark.sql.sources.parallelPartitionDiscovery.threshold Spark sparkdefaults.conf spark.sql.statistics.fallBackToHdfs Spark sparkdefaults.conf spark.sql.storeAssignmentPolicy Spark sparkdefaults.conf spark.sql.subquery.maxThreadThreshold Spark sparkdefaults.conf spark.sql.support.block.inferior.sql Spark sparkdefaults.conf spark.storage.decommission.shuffleBlocks.maxThreads Spark sparkdefaults.conf spark.task.reaper.enabled Spark sparkdefaults.conf spark.unsafe.sorter.spill.reader.buffer.size Spark sparkdefaults.conf spark.yarn.appMasterEnv.JAVAHOME Spark sparkdefaults.conf spark.yarn.containerLauncherMaxThreads Spark sparkdefaults.conf spark.yarn.scheduler.heartbeat.intervalms Spark sparkdefaults.conf spark.yarn.scheduler.initialallocation.interval Spark sparkenv.sh customsparkenv Spark sparkenv.sh sparkengine YARN mapredsite.xml mapreduce.application.classpath YARN mapredsite.xml mapreduce.cluster.acls.enabled YARN mapredsite.xml mapreduce.framework.name YARN mapredsite.xml mapreduce.job.aclmodifyjob YARN mapredsite.xml mapreduce.job.counters.counter.name.max YARN mapredsite.xml mapreduce.job.counters.group.name.max YARN mapredsite.xml mapreduce.job.counters.groups.max YARN mapredsite.xml mapreduce.job.counters.max YARN mapredsite.xml mapreduce.jobhistory.admin.acl YARN mapredsite.xml mapreduce.jobhistory.bindhost YARN mapredsite.xml mapreduce.jobhistory.donedir YARN mapredsite.xml mapreduce.jobhistory.http.policy YARN mapredsite.xml mapreduce.jobhistory.intermediatedonedir YARN mapredsite.xml mapreduce.jobhistory.recovery.enable YARN mapredsite.xml mapreduce.jobhistory.recovery.store.leveldb.path YARN mapredsite.xml mapreduce.map.env YARN mapredsite.xml mapreduce.map.java.opts YARN mapredsite.xml mapreduce.map.log.level YARN mapredsite.xml mapreduce.map.memory.mb YARN mapredsite.xml mapreduce.map.output.compress YARN mapredsite.xml mapreduce.map.output.compress.codec YARN mapredsite.xml mapreduce.map.sort.spill.percent YARN mapredsite.xml mapreduce.map.speculative YARN mapredsite.xml mapreduce.output.fileoutputformat.compress YARN mapredsite.xml mapreduce.output.fileoutputformat.compress.codec YARN mapredsite.xml mapreduce.reduce.env YARN mapredsite.xml mapreduce.reduce.input.buffer.percent YARN mapredsite.xml mapreduce.reduce.java.opts YARN mapredsite.xml mapreduce.reduce.log.level YARN mapredsite.xml mapreduce.reduce.memory.mb YARN mapredsite.xml mapreduce.reduce.shuffle.fetch.retry.enabled YARN mapredsite.xml mapreduce.reduce.shuffle.fetch.retry.intervalms YARN mapredsite.xml mapreduce.reduce.shuffle.fetch.retry.timeoutms YARN mapredsite.xml mapreduce.reduce.shuffle.input.buffer.percent YARN mapredsite.xml mapreduce.reduce.shuffle.merge.percent YARN mapredsite.xml mapreduce.reduce.shuffle.parallelcopies YARN mapredsite.xml mapreduce.reduce.speculative YARN mapredsite.xml mapreduce.shuffle.port YARN mapredsite.xml mapreduce.task.io.sort.factor YARN mapredsite.xml mapreduce.task.io.sort.mb YARN mapredsite.xml mapreduce.task.timeout YARN mapredsite.xml yarn.app.mapreduce.am.env YARN mapredsite.xml yarn.app.mapreduce.am.log.level YARN mapredsite.xml yarn.app.mapreduce.am.resource.mb YARN mapredsite.xml yarn.app.mapreduce.am.stagingdir YARN yarnsite.xml hadoop.http.authentication.simple.anonymous.allowed YARN yarnsite.xml hadoop.http.filter.initializers YARN yarnsite.xml hadoop.registry.client.auth YARN yarnsite.xml yarn.acl.enable YARN yarnsite.xml yarn.logaggregation.retainseconds YARN yarnsite.xml yarn.logaggregationenable YARN yarnsite.xml yarn.nodelabels.enabled YARN yarnsite.xml yarn.nodelabels.fsstore.rootdir YARN yarnsite.xml yarn.nodemanager.address YARN yarnsite.xml yarn.nodemanager.containerexecutor.class YARN yarnsite.xml yarn.nodemanager.diskhealthchecker.maxdiskutilizationperdiskpercentage YARN yarnsite.xml yarn.nodemanager.localizer.cache.targetsizemb YARN yarnsite.xml yarn.nodemanager.localizer.client.threadcount YARN yarnsite.xml yarn.nodemanager.localizer.fetch.threadcount YARN yarnsite.xml yarn.nodemanager.log.retainseconds YARN yarnsite.xml yarn.nodemanager.logaggregation.compressiontype YARN yarnsite.xml yarn.nodemanager.logaggregation.debugenabled YARN yarnsite.xml yarn.nodemanager.logaggregation.numlogfilesperapp YARN yarnsite.xml yarn.nodemanager.logaggregation.rollmonitoringintervalseconds YARN yarnsite.xml yarn.nodemanager.recovery.dir YARN yarnsite.xml yarn.nodemanager.recovery.enabled YARN yarnsite.xml yarn.nodemanager.recovery.supervised YARN yarnsite.xml yarn.nodemanager.remoteapplogdir YARN yarnsite.xml yarn.nodemanager.remoteapplogdirsuffix YARN yarnsite.xml yarn.nodemanager.resource.cpuvcores YARN yarnsite.xml yarn.nodemanager.resource.memorymb YARN yarnsite.xml yarn.nodemanager.resource.percentagephysicalcpulimit YARN yarnsite.xml yarn.nodemanager.resourcemanager.connect.wait.secs YARN yarnsite.xml yarn.nodemanager.vmemcheckenabled YARN yarnsite.xml yarn.nodemanager.vmempmemratio YARN yarnsite.xml yarn.nodemanager.webapp.crossorigin.enabled YARN yarnsite.xml yarn.resourcemanager.clusterid YARN yarnsite.xml yarn.resourcemanager.fusing.enable YARN yarnsite.xml yarn.resourcemanager.fusingmaxapigetjobs YARN yarnsite.xml yarn.resourcemanager.ha.rmids YARN yarnsite.xml yarn.resourcemanager.hostname.rm1 YARN yarnsite.xml yarn.resourcemanager.hostname.rm2 YARN yarnsite.xml yarn.resourcemanager.maxcompletedapplications YARN yarnsite.xml yarn.resourcemanager.recovery.enabled YARN yarnsite.xml yarn.resourcemanager.scheduler.autocorrect.container.allocation YARN yarnsite.xml yarn.resourcemanager.scheduler.class YARN yarnsite.xml yarn.resourcemanager.scheduler.monitor.enable YARN yarnsite.xml yarn.resourcemanager.store.class YARN yarnsite.xml yarn.resourcemanager.webapp.address.rm1 YARN yarnsite.xml yarn.resourcemanager.webapp.address.rm2 YARN yarnsite.xml yarn.resourcemanager.webapp.crossorigin.enabled YARN yarnsite.xml yarn.scheduler.maximumallocationmb YARN yarnsite.xml yarn.scheduler.maximumallocationvcores YARN yarnsite.xml yarn.scheduler.minimumallocationmb YARN yarnsite.xml yarn.scheduler.minimumallocationvcores YARN yarnsite.xml yarn.timelineservice.client.besteffort YARN yarnsite.xml yarn.timelineservice.client.maxretries YARN yarnsite.xml yarn.timelineservice.enabled YARN yarnsite.xml yarn.timelineservice.httpcrossorigin.enabled YARN yarnsite.xml yarn.webapp.ui2.enable