大数据基础之集群搭建
Cluster OS&Platform
redhat/centos7, docker, mesos, cloudera manager(cdh)
Checklist
1 check user & password & network reachability, make sure everything is fine to login all remote servers by ssh client
2 check linux release, upgrade or reinstall if necessary
3 check yum repo
4 check hardware: memory, cpu, disk & partition, network device
5 check network: lan & wlan environment
5 check data disk, make sure all data disks are mounted by fdisk/mke2fs/mount, /etc/fstab
6 check user & group, running processes, free memory, free disk, directory structure
6 check iptables, make sure only the necessary protocols and ports are allowed
7 check systemd, service
Preparation
1 login the first server by ssh client, it will be acted as deploy center
2 upload 'deploy' directory(more than 15G: iso8G, cdh3G,registry2G) and 'deploy.sh' to $HOME by sftp or other tools
3 use deploy.sh to generate deploy commands
4 deploy the cluster offline
Offline Deployment Details
1 make sure root user login without password by ssh client between all servers
2 create yum repo file using iso on local server
3 install ansible on local server
4 create & push /etc/hosts to all servers by ansible
5 install & start docker on local server
6 docker load registry image from local file
7 docker run registry on local server
8 docker run nginx on local server as 'yum repo server', 'docker registry server' and 'cloudera manager proxy server'
9 docker run mysql on local server temporarily
10 create mysql databases & users and flush privileges
11 create & push yum repo file to all servers by ansible
12 push deploy directory to all servers by ansible
13 install ansible on other servers
14 install and start docker on other servers
15 install kerberos on all servers
16 start kdc on local server
17 install jdk on all servers
18 install cloudera manager & cdh on all servers
19 start cloudera manager anget on all servers
20 start cloudera manager master on local server
21 deploy zookeeper by cloudera manager
22 install mesos and marathon on all servers
23 start mesos agent on all servers
24 start mesos master on other two servers
25 start marathon master on other two servers
26 deploy marathon-lb on marathon by api
27 docker stop mysql and redeploy mysql master&slave on marathon by api
28 deploy redis, elasticsearch, logstash, kibana, kafka, airflow on marathon by api
29 deploy hdfs, yarn, hive, hbase, oozie, kudu, impala, sentry, hue by cloudera manager
30 enable kerberos on cloudera manager
31 deploy all kinds of client
32 confirm all components working
33 initialize data structure in mysql, hive, es, impala
34 deploy business components like tomcat, logstash on marathon
35 trigger all crontab stuff in airflow
36 confirm all business working
Optional
use zabbix to monitor all servers
相关文章
- 零基础搭建k8s集群一次成功,坑太多我都替你踩过了(文档)
- 上篇:docker基础玩转
- win32 多线程基础
- Java 基础学习(9)
- R语言基础
- Linux基础:Linux NFS机制工作原理及解析
- javaweb学习总结(十五)——JSP基础语法详解编程语言
- Oracle Q 操作使你快速掌握关系数据库基础(oracle q 操作)
- 免费体验Oracle云上基础服务(oracle云免费基础服务)
- Oracle 会话状态系统稳定性的基础(oracle 会话状态)
- 从零开始Redis入门学习指南(redis零基础学习)
- 入门Redis 从零开始学习(redis 零基础)
- 基础:在 Linux 终端中删除文件和文件夹