YARN（MRv2）搭建</property> <property> <

文件名称: YARN（MRv2）搭建

所属分类: Hadoop

开发工具:

文件大小: 302kb

下载次数: 0

上传时间: 2019-07-02

提供者: aba****

下载 (302kb)

不能下载？报告错误

详细说明：YARN（MRv2）搭建 yarn resourcemanager address master004: 8032 property> yarn resourcemanager scheduler address mastero04: 8030 property> yarn, resourcemanager, admin. address/name> master004: 8033 /property yarn, resourcemanager, webapp. address master004: 8088 yarn, nodemanager. aux-services s/name> mapreduce_ shuffle property> yarn. nodemanager, aux-services. mapreduce_shuffle. class org. apache hadoop. mapred ShuffleHandler yarn. log-aggregation-enable true List of directories to store localized files in yarn. nodemanager local-dirs /data/disk01/hadoop/yarn/local, /data/disk02/hadoop/yarn/local /data/disk03/hadoop/yarnlocal, /data/disk04/hadoop/yarn/local /data/disk05/hadoop/yarnlocal yarn, nodemanager log-dirs /data/disk0 1/hadoopyarn/logs/data/disko/hadoop/yarn/logs, /data/disk03/hadoop/yarn/logs,/data/disk04/hadoop/yarn/logs /data/disk05/hadoop/yarn/ogs property> Where to aggregate logs to yarn, nodemanager remote-app-log-dirs/name> /var/log/hadoop-yarn/apps yarn application. classpath SHADOOP CONF DIR SHADOOP COMMON HOME/, $HADOOP COMMON HOME/ib/ SHADOOP_ HD FS_HOME/ $HADOOP__HD FS_ HOME/ib/ SHADOOP_ MAPRED HOME/*$HADOOP MAPRED HOME/ib/ SHADOOP YARN HOME/*, $HADOOP YARN HOME/lib/ /property> yarn. app. mapreduce. am. staging-dir /users/value> property> The minimum allocation for every container request at the rm in MBs. Memory requests lower than this won't take effect and the specified value will get allocated at minimum. yarn scheduler. minimum-allocation-mb 1024 The maximum allocation for every container request at the rm in MBs. Memory requests higher than this won't take effect and will get capped to this value. yarn scheduler. maximum-allocation-mb 16384 property> yarn, scheduler. minimum-allocation-vcoress/name> 1 The maximum allocation for every container request at the rM in terms of virtual CPU cores. Requests higher than this wont take effect and will get capped to this value. yarn scheduler. maximum-allocation-vcores 32 Number of CPU cores that can be allocated for containers. yarn. nodemanager resource cpu-vcores 48 Amount of physical memory in MB, that can be allocated for containers. yarn. nodemanager resource memory-mb 120000 Ratio between virtual memory to physical memory when setting memory limits for containers. Container allocations are expressed in terms of physical memory and virtual memory usage is allowed to exceed this allocation by this ratio yarn. nodemanager, vmem-pmem-ratios/name> 6 YARN配置 1 nodemanager创建本地目录 mkdir -p/data/disk01/hadoop/yarn/local /data/disk02/hadoop/yarn/loca /data/disk03/hadoop/yarn/local /data/disk04/ hadoop/yarn/local /data/disko/ hadoop/yarn/local mkdir -p/data/disk0 1/hadoop/yarn/logs/ data/disk02/hadoop/yarn/logs /data/disk03/hadoop/yarn/logs /data/disk04/hadoop/yarn/logs /data/disk05/hadoop/yarn/logs chown -R yarn: yarn /data/disko 1/hadoop/yarn /data/disk02/hadoop/yarn/local /data/disk03/hadoop/yarn /data/disk04/hadoop/yarn /data/disk05/hadoop/yarn chown -R yarn: yarn/data/disko1/ hadoop/yarn/local /data/ disk02/hadoop/yarn/local /data/disk03/ hadoop/yarn/local /data/disk04/hadoop/ yarn/loca /data/disk05/hadoop/yarn/loca chown -R yarn: yarn /data/disko1/hadoop/yarn/logs /data/disk02/ hadoop/yarn/logs /data/disk03/hadoop/yarn/logs /data/disk04/hadoop/yarn/logs /data/disk05/hadoop/yarn/logs 2创建 historyl目录 sudo-u hdfs hadoop fs -mkdir /user/history sudo -u hdfs hadoop fs -chmod -R 1777/user/history sudo -u hdfs hadoop fs -chown yarn /user/history 3启动服务 resourcemanager sudo service hadoop-yarn-resourcemanager start nodemanager sudo service hadoop-yarn-nodemanager start mapreduce-historyserver: sudo service hadoop-mapreduce-historyserver start 运行示例 hadoop jar /usr/lib/hadoop-mapreduce hadoop-mapreduce-examples,jar wordcount Mapreduce map, memory mb=4096 input wcout Yarn运维安装环境安装路径 /usr/l ib/hadoop-yarn 配置文件路径 /elc/hadoop/conf 日志路径 /var/log/hadoop-yarn 运行|关闭|查看状态 resourcemanager service hadoop-yarn-resourcemanager start stop status nodemanager service hadoop yarn- nodemanager start stop status mapreduce-historyserver service hadoop-mapreduce-historyserver start stop status 常用命令查看节点状态 yarn nlode -list -all resourcemanager管理 yarm rmadmin