大数据平台CDH6.3.2部署
标签（空格分隔）： 大数据平台构建
- 一： 环境初始化
- 二： 安装CDH6.3.2
一：环境初始化
1.1 环境介绍
系统：CentOS7.5X64
cat /etc/hosts
----
192.168.11.160 dev01.lanxintec.cn
192.168.11.161 dev02.lanxintec.cn
192.168.11.162 dev03.lanxintec.cn
----
1.2：无密钥登录配置
做root用户无密钥认证
ssh-keygen ---一直敲回车到最后
cat id_rsa.pub >> authorized_keys
chmod 600 authorized_keys
将所有的公钥导入authorized_keys 分发到 所有的 机器的.ssh/ 下面
然后测试
1.3:关闭 firewalld，iptables 与 selinux
所有节点都要配置：
yum install -y firewall* iptable*
(1) 关闭firewalld
systemctl stop firewalld.service
systemctl disable firewalld.service
firewall-cmd --state
(2) 关闭iptables
iptables -F
systemctl stop iptables.service
service iptables save
systemctl disable iptables.service
(3) 禁用 selinux
sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config
setenforce 0
getenforce 0
sestatus
重启机器
reboot
2.4：所有机器更改CentOS7.5x64的文件句柄数
vim /etc/security/limits.conf
---
* soft nofile 65535
* hard nofile 1029345
* soft nproc unlimited
* hard nproc unlimited
* soft memlock unlimited
* hard memlock unlimited
在其它 节点上面执行命令：
rsync -arvzP root@192.168.100.11:/etc/security/limits.conf /etc/security/
同步limits.conf 文件
2.5 时间同步：
所有机器 yum install chrony*
1. dev01.lanxintec.cn---dev03.lanxintec.cn 同步阿里云的时间服务器:
vim /etc/chrony.conf 增加
----
注释掉相关默认的server
server ntp1.aliyun.com iburst
service chronyd start
chkconfig chronyd on
service chronyd stop
service chronyd start
chkconfig chronyd on
2.6: cdh6.3.2 安装所需文件下载
cdh6.3.2 离线包：
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el6.parcel.sha256
manifest.json
下载： https://archive.cloudera.com/cdh6/6.3.2/parcels/
cdh6.3.1 的CM 包:
cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm
oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
下载地址：
https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/
jdbc 驱动：
http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.46/mysql-connector-java-5.1.46.jar
http://archive.cloudera.com/gplextras/misc/ext-2.2.zip
2.7:安装jdk
所用大数据节点都要配置：jdk
rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
vim /etc/profile
---
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
---
source /etc/profile
java -version
2.8 安装mariadb-server
yum install -y mariadb-server
service mariadb start
chkconfig mariadb on
mysql_secure_installation
mysql -uroot -pflyfish225
2.9配置 httpd 分发服务器
pre01.lanxintec.cn:
yum install -y httpd
vim /etc/httpd/conf/httpd.conf
----
修改/etc/httpd/conf/httpd.conf配置文件，在<IfModule mime_module>中修改以下内容
AddType application/x-gzip .gz .tgz .parcel
----
service httpd start
chkconfig httpd start
做CM6.3.1 的 分发源
yum install -y createrepo
cd /var/www/html/cdh5.3.6
createrepo.
vim /etc/yum.repos.d/cloduera-manger.repo
------
[cmrepo]
name = cm_repo
baseurl =http://192.168.100.11/cdh6
enable = 1
gpgcheck = 0
-----
yum makecache fast
所有主机执行命令同步：
rsync -avrzP root@192.168.11.160:/etc/yum.repos.d/cloudera-manager.repo /etc/yum.repos.d/
yum makecache fast
2.10 配置mysql-jdbc
cd /usr/share/java/
cp -p mysql-connector-java-8.0.15.jar mysql-connector-java.jar
三： 安装CDH6.3.2所需环境
3.1:安装所需依赖包
所有大数据节点都要安装：
yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb postgresql* portmap mod_ssl openssl-devel python-psycopg2 MySQL-python
3.2 安装cloudera-server
yum install -y cloudera-manager-server
3.3 在 mysql 当中创建 CM 需要的注册库
# mysql -uroot -pflyfish225
mysql> create database cmf character set utf8;
mysql> grant all privileges on cmf.* to 'cmf'@'%' identified by 'cmf_1234' with grant option;
mysql> grant all privileges on cmf.* to 'cmf'@'dev01.lanxintec.cn' identified by 'cmf_1234' with grant option;
mysql> flush privileges;
3.4 注入CM-server库 与 CM-agent端的server
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql cmf cmf cmf_1234
四： 配置CDH6.3.2的cloudera-manager安装
4.1 安装启动CM
service cloudera-scm-server start
cd /var/log/cloudera-scm-server/
tail -f cloudera-scm-server.log
五： 打开web 安装：
echo 10 > /proc/sys/vm/swappiness
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo "vm.swappiness = 10" >> /etc/sysctl.conf
sysctl -p
vim /etc/rc.local
----
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
---
create database cdh_hive default character set latin1;
grant all privileges on cdh_hive.* to 'hive'@'%' identified by 'Hive_123' with grant option;
create database cdh_oozie default character set utf8;
grant all privileges on cdh_oozie.* to 'oozie'@'%' identified by 'Oozie_123' with grant option;
create database cdh_hue default character set utf8;
grant all privileges on cdh_hue.* to 'hue'@'%' identified by 'Hue_123' with grant option;
create database cdh_rm default character set utf8;
grant all privileges on cdh_rm.* to 'rm'@'%' identified by 'Rm_123' with grant option;
flush privileges;
