Day 29 Cloudera Manager

CDH 5.16.2 Deploy

Cloudera Manager

载点

https://docs.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_516.html

网路设定

  1. 设定hostname
hostnamectl set-hostname manager-1.example.com
  1. 编辑/etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

10.0.1.239  manager-1.example.com  manager-1
10.0.1.240  worker-1.example.com  worker-1
10.0.1.241  worker-2.example.com  worker-2
10.0.1.238  worker-3.example.com  worker-3

安装 JDK

yum update
yum install -y java-1.8.0-openjdk-devel

安装 Cloudera Manager Server

下载cloudera-manager repository

yum install -y wget
  1. 下载Cloudera-manager repository 至 /etc/yum.repos.d/ 
wget https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.d/
  1. Import the repository signing GPG key & yum install
rpm --import https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera
  1. yum install
yum install -y cloudera-manager-daemons cloudera-manager-server

设定Database

install & init db

yum install -y postgresql-server
sudo su -l postgres -c "postgresql-setup initdb"

编辑 /var/lib/pgsql/data/pg_hba.conf, 设定 MD5 authentication

host all all 127.0.0.1/32 md5

编辑 /var/lib/pgsql/data/postgresql.conf

listen_addresses = '*'
max_connection = 100
shared_buffers = 256MB
wal_buffers = 8MB
checkpoint_segments = 16
checkpoint_completion_target = 0.9

启用postgreSQL

systemctl enable postgresql
systemctl restart postgresql

Creating Databases for Cloudera Software

  1. Connect to PostgreSQL:
sudo -u postgres psql
  1. Create databases
postgres=# CREATE ROLE scm LOGIN PASSWORD 'scm';
postgres=# CREATE DATABASE scm OWNER scm ENCODING 'UTF8';
postgres=# CREATE ROLE amon LOGIN PASSWORD 'amon';
postgres=# CREATE DATABASE amon OWNER amon ENCODING 'UTF8';
postgres=# CREATE ROLE rman LOGIN PASSWORD 'rman';
postgres=# CREATE DATABASE rman OWNER rman ENCODING 'UTF8';
postgres=# CREATE ROLE nav LOGIN PASSWORD 'nav';
postgres=# CREATE DATABASE nav OWNER nav ENCODING 'UTF8';
postgres=# CREATE ROLE navms LOGIN PASSWORD 'navms';
postgres=# CREATE DATABASE navms OWNER navms ENCODING 'UTF8';
  1. 设定Cloudera Manager Database
sudo /usr/share/cmf/schema/scm_prepare_database.sh postgresql scm scm

启用Cloudera Manager Server

  1. Start Cloudera Manager Server
systemctl start cloudera-scm-server
  1. 查看cloudera-scm-server.log 
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

看到以下log讯息表示cloudera-scm-server已启动完成
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

  1. 到web介面 http://<manager_server_host>:7180 登入
  • Username: admin
  • Password: admin

Cloudera Agent(worker node for cluster)

网路设定

  • 设定hostname
hostnamectl set-hostname worker-1.example.com

安装 JDK

yum update
yum install -y java-1.8.0-openjdk-devel

安装 cloudera-manager-agent

下载cloudera-manager repository

yum install -y wget
  1. 下载Cloudera-manager repository 至 /etc/yum.repos.d/ 
wget https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.d/
  1. Import the repository signing GPG key
rpm --import https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera

下载Cloudera-manager-agent

  1. install the package
yum install -y cloudera-manager-daemons cloudera-manager-agent
  1. 编辑 /etc/cloudera-scm-agent/config.ini，设定Manager的host, port
server_host=<manager-node ip> 
server_port=7182
  1. 启用 cloudera-manager-agent
systemctl start cloudera-scm-agent

问题解决

启动丛集

yarn

  • 错误讯息

IO error:/var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/LOCK: Permission denied

  1. 到NodeManager role的节点
  2. 更改/var/lib/hadoop-yarn权限
  3. chmod 755 /var/lib/hadoop-yarn

hue load balancer start failed

  • 错误讯息

/run/cloudera-scm-agent/process/216-hue-HUE_LOAD_BALANCER/httpd.conf is invalid

  1. 到hue LoadBalancer role的节点
  2. 安装 httpd, mod_ssl 
yum install -y httpd
yum install -y mod_ssl
systemctl start httpd
  1. 若还是失败，可能是seLinux设定必须关闭
  2. 编辑/etc/sysconfig/selinux 
SELINUX=disabled
SELINUXTYPE=targeted
  1. 重启LoadBalancer

zookeeper start failed

  1. 到zookeeper role 的节点
  2. 删除 /var/lib/zookeeper/version-2
  3. 重新建立 /var/lib/zookeeper/version-2，修改用户
mkdir /var/lib/zookeeper/version-2
chown zookeeper:zookeeper /var/lib/zookeeper/version-2
chmod 755 /var/lib/zookeeper/version-2
  1. 重启zookeepr

oozie start failed

  1. 到oozie role 的节点
  2. 修改/var/lib/oozie权限及用户
chown oozie:oozie /var/lib/oozie
chmod 755 /var/lib/oozie
  1. 重启oozie

<<:  HERE mSDK - Map Rendering

>>:  使用Fortigate的DoS功能

树选手2号：random forest [python实例]

今天来用前几天使用判断肿瘤良性恶性的例子来执行random forest，一开始我们一样先建立sco...

【第二十九天 - 系统分析 题目分析】

先简单回顾一下，今天预计分析的题目： 题目连结：https://leetcode.com/prob...

第二十七天：脱壳又称脱衣服?!

大家好我是Andy，今天来到了第二十七天，昨天讲述了加壳的技巧以及种种的工具，今天我们就来讲一下脱壳...

[13th][Day5] Goroutine

about goroutine 要聊 golang 中的 「go」 这个保留字之前，先来说说 Gor...

Day8 主动情蒐-情蒐流程、工具与漏洞简介

针对主动情蒐 主动情蒐指主动跟目标进行互动，会透过工具枚举所需的资讯，本系列文会介绍以下几种工具。 ...