Linux下部署Hadoop伪分布模式
2023-09-11 14:20:33 时间
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory /home/hadoop/.ssh.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Passphrases do not match. Try again.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
2f:a9:60:c7:dc:38:8f:c7:bb:70:de:d4:39:c3:39:87 hadoop@promote.cache-dns.local
The keys randomart image is:
+--[ RSA 2048]----+
| |
| |
| |
| |
| S |
| o o o o + |
| o B.= o E . |
| . o Oo+ = |
| o.=o. |
+-----------------+
最终将在/home/hadoop/.ssh/路径下生成私钥id_rsa和公钥id_rsa.pub [hadoop@promote .ssh]$ cd /home/hadoop/.ssh/ [hadoop@promote .ssh]$ ls id_rsa id_rsa.pub 修改sshd服务配置文件: [hadoop@promote .ssh]$ su root [root@promote .ssh]# vi /etc/ssh/sshd_config 启用RSA加密算法验证(去掉前面的#号)
PubkeyAuthentication yes # The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2 # but this is overridden so installations will only check .ssh/authorized_keys AuthorizedKeysFile .ssh/authorized_keys
修改mapred-site.xml:
[hadoop@promote bin]$ sh start-all.sh starting namenode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-namenode-promote.cache-dns.local.out localhost: starting datanode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-promote.cache-dns.local.out localhost: starting secondarynamenode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-secondarynamenode-promote.cache-dns.local.out starting jobtracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-jobtracker-promote.cache-dns.local.out localhost: starting tasktracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-promote.cache-dns.local.out
[root@promote .ssh]# service sshd restart Redirecting to /bin/systemctl restart sshd.service [root@promote .ssh]# ps -ef|grep sshd root 1995 1 0 22:33 ? 00:00:00 sshd: hadoop [priv] hadoop 2009 1995 0 22:33 ? 00:00:00 sshd: hadoop@pts/0 root 4171 1 0 23:11 ? 00:00:00 /usr/sbin/sshd -D root 4175 3397 0 23:12 pts/0 00:00:00 grep --color=auto sshd
然后切换回hadoop用户,将ssh证书公钥拷贝至/home/hadoop/.ssh/authorized_keys文件中 [root@promote .ssh]# su hadoop [hadoop@promote .ssh]$ cat id_rsa.pub authorized_keys 修改authorized_keys文件的权限为644(这步一定要有) [hadoop@promote .ssh]$ chmod 644 authorized_keys [hadoop@promote .ssh]$ ssh localhost The authenticity of host localhost (127.0.0.1) cant be established. RSA key fingerprint is 25:1f:be:72:7b:83:8e:c7:96:b6:71:35:fc:5d:2e:7d. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added localhost (RSA) to the list of known hosts. Last login: Thu Feb 13 23:42:43 2014 第一次登陆将会将证书内容保存在/home/hadoop/.ssh/known_hosts文件中,以后再次登陆将不需要输入密码 [hadoop@promote .ssh]$ ssh localhost Last login: Thu Feb 13 23:46:04 2014 from localhost.localdomain 至此ssh证书部分配置完成 第二步:安装JDK [hadoop@promote ~]$ java -version java version "1.7.0_25" OpenJDK Runtime Environment (fedora-2.3.10.3.fc19-i386) OpenJDK Client VM (build 23.7-b01, mixed mode) 将OpenJDK换为Oracle的Java SE [hadoop@promote .ssh]$ cd ~ [hadoop@promote ~]$ uname -i i386 在Oracle的官网下载jdk-6u45-linux-i586.bin后上传至服务器,赋予权限并进行安装,最后删除安装包 [hadoop@promote ~]$ chmod u+x jdk-6u45-linux-i586.bin [hadoop@promote ~]$ ./jdk-6u45-linux-i586.bin [hadoop@promote ~]$ rm -rf jdk-6u45-linux-i586.bin [hadoop@promote conf]$ export PATH=$PATH:/home/hadoop/jdk1.6.0_45/bin 出现以下结果说明JDK成功安装: [hadoop@promote ~]$ java -version java version "1.6.0_45" Java(TM) SE Runtime Environment (build 1.6.0_45-b06) Java HotSpot(TM) Client VM (build 20.45-b01, mixed mode, sharing) 第三步:安装Hadoop 在Hadoop官网下载hadoop-1.2.1.tar.gz并上传至服务器/home/hadoop路径下 [hadoop@promote ~]$ tar -xzf hadoop-1.2.1.tar.gz [hadoop@promote ~]$ rm -rf hadoop-1.2.1.tar.gz [hadoop@promote ~]$ cd hadoop-1.2.1/conf/ [hadoop@promote conf]$ vi hadoop-env.sh 将JAVA_HOME指向第二步安装的JDK所在目录片 # The java implementation to use. Required. export JAVA_HOME=/home/hadoop/jdk1.6.0_45 保存并退出 第四步:修改Hadoop配置文件 修改core-site.xml:
保姆级丨Hadoop部署 Hadoop是一个由Apache基金会所开发的分布式系统基础架构。用户可以在不了解分布式底层细节的情况下,开发分布式程序。充分利用集群的威力进行高速运算和存储。
最终将在/home/hadoop/.ssh/路径下生成私钥id_rsa和公钥id_rsa.pub [hadoop@promote .ssh]$ cd /home/hadoop/.ssh/ [hadoop@promote .ssh]$ ls id_rsa id_rsa.pub 修改sshd服务配置文件: [hadoop@promote .ssh]$ su root [root@promote .ssh]# vi /etc/ssh/sshd_config 启用RSA加密算法验证(去掉前面的#号)
PubkeyAuthentication yes # The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2 # but this is overridden so installations will only check .ssh/authorized_keys AuthorizedKeysFile .ssh/authorized_keys
修改mapred-site.xml:
[hadoop@promote bin]$ sh start-all.sh starting namenode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-namenode-promote.cache-dns.local.out localhost: starting datanode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-promote.cache-dns.local.out localhost: starting secondarynamenode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-secondarynamenode-promote.cache-dns.local.out starting jobtracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-jobtracker-promote.cache-dns.local.out localhost: starting tasktracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-promote.cache-dns.local.out
[root@promote .ssh]# service sshd restart Redirecting to /bin/systemctl restart sshd.service [root@promote .ssh]# ps -ef|grep sshd root 1995 1 0 22:33 ? 00:00:00 sshd: hadoop [priv] hadoop 2009 1995 0 22:33 ? 00:00:00 sshd: hadoop@pts/0 root 4171 1 0 23:11 ? 00:00:00 /usr/sbin/sshd -D root 4175 3397 0 23:12 pts/0 00:00:00 grep --color=auto sshd
然后切换回hadoop用户,将ssh证书公钥拷贝至/home/hadoop/.ssh/authorized_keys文件中 [root@promote .ssh]# su hadoop [hadoop@promote .ssh]$ cat id_rsa.pub authorized_keys 修改authorized_keys文件的权限为644(这步一定要有) [hadoop@promote .ssh]$ chmod 644 authorized_keys [hadoop@promote .ssh]$ ssh localhost The authenticity of host localhost (127.0.0.1) cant be established. RSA key fingerprint is 25:1f:be:72:7b:83:8e:c7:96:b6:71:35:fc:5d:2e:7d. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added localhost (RSA) to the list of known hosts. Last login: Thu Feb 13 23:42:43 2014 第一次登陆将会将证书内容保存在/home/hadoop/.ssh/known_hosts文件中,以后再次登陆将不需要输入密码 [hadoop@promote .ssh]$ ssh localhost Last login: Thu Feb 13 23:46:04 2014 from localhost.localdomain 至此ssh证书部分配置完成 第二步:安装JDK [hadoop@promote ~]$ java -version java version "1.7.0_25" OpenJDK Runtime Environment (fedora-2.3.10.3.fc19-i386) OpenJDK Client VM (build 23.7-b01, mixed mode) 将OpenJDK换为Oracle的Java SE [hadoop@promote .ssh]$ cd ~ [hadoop@promote ~]$ uname -i i386 在Oracle的官网下载jdk-6u45-linux-i586.bin后上传至服务器,赋予权限并进行安装,最后删除安装包 [hadoop@promote ~]$ chmod u+x jdk-6u45-linux-i586.bin [hadoop@promote ~]$ ./jdk-6u45-linux-i586.bin [hadoop@promote ~]$ rm -rf jdk-6u45-linux-i586.bin [hadoop@promote conf]$ export PATH=$PATH:/home/hadoop/jdk1.6.0_45/bin 出现以下结果说明JDK成功安装: [hadoop@promote ~]$ java -version java version "1.6.0_45" Java(TM) SE Runtime Environment (build 1.6.0_45-b06) Java HotSpot(TM) Client VM (build 20.45-b01, mixed mode, sharing) 第三步:安装Hadoop 在Hadoop官网下载hadoop-1.2.1.tar.gz并上传至服务器/home/hadoop路径下 [hadoop@promote ~]$ tar -xzf hadoop-1.2.1.tar.gz [hadoop@promote ~]$ rm -rf hadoop-1.2.1.tar.gz [hadoop@promote ~]$ cd hadoop-1.2.1/conf/ [hadoop@promote conf]$ vi hadoop-env.sh 将JAVA_HOME指向第二步安装的JDK所在目录片 # The java implementation to use. Required. export JAVA_HOME=/home/hadoop/jdk1.6.0_45 保存并退出 第四步:修改Hadoop配置文件 修改core-site.xml:
保姆级丨Hadoop部署 Hadoop是一个由Apache基金会所开发的分布式系统基础架构。用户可以在不了解分布式底层细节的情况下,开发分布式程序。充分利用集群的威力进行高速运算和存储。
相关文章
- linux insmod *.ko Exec format error
- Linux内核函数调用规范(function call convention)
- Linux 挂载新硬盘
- linux安装包手记
- Linux:文件系统与日志分析
- 【Linux】/lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
- Linux自学篇——linux命令英文全称及解释
- 【Linux】Hadoop时钟同步
- 【Python】Linux Acanoda PySpark Spark
- Linux /proc/iomem与/proc/ioports
- 在Linux上编译Hadoop-2.4.0实践与总结
- 记录一次实验室linux系统的GPU服务器死机故障的排查——Linux系统的Power States
- Linux—linux 查看一个文件有多少M
- Linux_查看linux并发连接数
- linux服务器安装nodeJS步骤及踩坑记录(解决node -v报错cannot execute binary file: Exec format error的问题 - 在Linux中安装适用于arm64位的nodejs)以及node环境项目部署
- Linux上传文件时要注意的几点
- Linux scp命令详解
- 查询红帽linux/Oracle Linux的发行版本的方法
- linux小技巧(1)
- 【Linux】linux经常使用基本命令
- 【Linux】linux经常使用基本命令
- Linux学习笔记(18)Linux sed,linux awk,linux grep,Linux三剑客
- Linux_rsylogd日志轮替(三)
- Linux下汇编语言学习笔记50 ---