大数据基础之Doris(1)编译安装和启动
一 编译
doris编译有两种方式,一种是docker编译,一种是直接裸机编译,推荐使用docker编译,可以避免大量的环境依赖问题
docker编译
1 安装docker
yum install docker
systemctl start docker
systemctl enable docker
docker pull apachedoris/doris-dev:build-env-1.2
2 下载源码
wget https://mirrors.bfsu.edu.cn/apache/incubator/doris/0.13.0-incubating/apache-doris-0.13.0-incubating-src.tar.gz
tar xvf apache-doris-0.13.0-incubating-src.tar.gz
3 启动容器
docker run -it -v /root/.m2:/root/.m2 -v /path/to/apache-doris-0.13.0-incubating-src:/root/apache-doris-0.13.0-incubating-src apachedoris/doris-dev:build-env-1.2
需要做两个目录映射,一个是maven的repository目录,一个是doris源码目录,避免容器挂了之后之前下载或编译的内容丢失
4 编译doris
cd /root/apache-doris-0.13.0-incubating-src
sh -x build.sh
编译之后输出至output目录,有3个子目录:be、fe、udf,只需要拷贝output目录到其他服务器即可
编译时报错
[ERROR] Plugin net.sourceforge.czt.dev:cup-maven-plugin:1.6-cdh or one of its dependencies could not be resolved: Failed to read artifact descriptor for net.sourceforge.czt.dev:cup-maven-plugin:jar:1.6-cdh: Could not transfer artifact net.sourceforge.czt.dev:cup-maven-plugin:pom:1.6-cdh from/to spring-plugins (https://repo.spring.io/plugins-release/): Authentication failed for https://repo.spring.io/plugins-release/net/sourceforge/czt/dev/cup-maven-plugin/1.6-cdh/cup-maven-plugin-1.6-cdh.pom 401 Unauthorized -> [Help 1
修改如下:
<!-- for java-cup -->
<repository>
<!--
<id>cloudera-thirdparty</id>
<url>https://repository.cloudera.com/content/repositories/third-party/</url>
-->
<id>cloudera-public</id>
<url>https://repository.cloudera.com/artifactory/public/</url>
</repository>
<!-- for cup-maven-plugin -->
<pluginRepository>
<!--
<id>spring-plugins</id>
<url>https://repo.spring.io/plugins-release/</url>
-->
<id>cloudera-public</id>
<url>https://repository.cloudera.com/artifactory/public/</url>
</pluginRepository>
5 编译broker
cd fs_brokers/apache_hdfs_broker
sh -x build.sh
6 编译spark connector
cd extension/spark-doris-connector
sh -x build.sh
裸机编译
1 准备
jdk8+
maven
sudo yum groupinstall 'Development Tools' && sudo yum install maven cmake byacc flex automake libtool bison binutils-devel zip unzip ncurses-devel curl git wget python2 glibc-static libstdc++-static java-1.8.0-openjdk
其中:centos7上gcc默认4.8.5,cmake默认2.8.12
2 升级GCC
yum install gcc-c++
wget http://ftp.tsukuba.wide.ad.jp/software/gcc/releases/gcc-7.3.0/gcc-7.3.0.tar.gz
tar zxvf tar zxvf gcc-7.3.0.tar.gz
cd gcc-7.3.0
yum install lbzip2
./contrib/download_prerequisites
mkdir build
cd build/
../configure --enable-checking=release --enable-languages=c,c++ --disable-multilib
make
make install
3 升级CMAKE
wget https://cmake.org/files/v3.6/cmake-3.6.2.tar.gz
tar xvf cmake-3.6.2.tar.gz && cd cmake-3.6.2/
./bootstrap
gmake
gmake install
mv /usr/bin/cmake /usr/bin/cmake.bak
ln -s /usr/local/bin/cmake /usr/bin/
4 编译
sh -x build.sh
报错处理
报错1
Downloading libevent-20180622-24236aed01798303745470e6c498bf606e88724a.zip from https://doris-incubating-repo.bj.bcebos.com/thirdparty/libevent-20180622-24236aed01798303745470e6c498bf606e88724a.zip to /usr/local/app/doris/thirdparty/src
--2021-01-11 09:32:59-- https://doris-incubating-repo.bj.bcebos.com/thirdparty/libevent-20180622-24236aed01798303745470e6c498bf606e88724a.zip
Resolving doris-incubating-repo.bj.bcebos.com (doris-incubating-repo.bj.bcebos.com)... 2409:8c00:6c21:10ad:0:ff:b00e:67d, 220.181.33.44, 220.181.33.43
Connecting to doris-incubating-repo.bj.bcebos.com (doris-incubating-repo.bj.bcebos.com)|2409:8c00:6c21:10ad:0:ff:b00e:67d|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-01-11 09:32:59 ERROR 404: Not Found.
因为url已经修改,参考git上最新的文件如下:
https://github.com/apache/incubator-doris/blob/master/thirdparty/vars.sh
解决方法:将url修改为
LIBEVENT_DOWNLOAD="https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/libevent-20180622-24236aed01798303745470e6c498bf606e88724a.zip"
参考:https://github.com/apache/incubator-doris/issues/5519
报错2
下载第三方DataTables报错403,直接访问
https://datatables.net/download/builder?bs-3.3.7/jq-3.3.1/dt-1.10.22
提示
Error: Only libraries of the current release version can be used by the. package builder. The library DataTables's current release version is 1.10.24. Please reload the download builder page to have it use the latest libraries.
解决方法:修改thirdparty/vars.sh,将版本改为最新的1.10.24,同时修改md5sum
# datatables, bootstrap 3 and jQuery 3
#DATATABLES_DOWNLOAD="https://datatables.net/download/builder?bs-3.3.7/jq-3.3.1/dt-1.10.22"
#DATATABLES_NAME="DataTables.zip"
#DATATABLES_SOURCE="DataTables-1.10.22"
#DATATABLES_MD5SUM="62558846fc6a6db1428e7816a2a351f7"
DATATABLES_DOWNLOAD="https://datatables.net/download/builder?bs-3.3.7/jq-3.3.1/dt-1.10.24"
DATATABLES_NAME="DataTables.zip"
DATATABLES_SOURCE="DataTables-1.10.24"
DATATABLES_MD5SUM="22404292d02cf3c5f4cd9f5a02d4b42c"
报错3
checking how to run the C preprocessor... /usr/lib64/ccache/../bin/cpp
configure: error: in `/data/app/apache-doris-0.13.0-incubating-src/thirdparty/src/unixODBC-2.3.7':
configure: error: C preprocessor "/usr/lib64/ccache/../bin/cpp" fails sanity check
解决方法:
ln -s /usr/local/bin/cpp /usr/lib64/bin/cpp
报错4
./comp_err: error while loading shared libraries: libatomic.so.1: cannot open shared object file: No such file or directory
解决方法:
yum -y install libatomic
报错5
cp: cannot stat ‘./zstd_ep-install/lib/libzstd.a’: No such file or directory
解决方法:
cd thirdparty/src/zstd-1.3.7/tests
make zstd-staticLib
mkdir -p /path/to/apache-doris-0.12.0-incubating-src/thirdparty/src/arrow-apache-arrow-0.15.1/cpp/release/zstd_ep-install/lib64
cp ../lib/libzstd.a /path/to/apache-doris-0.12.0-incubating-src/thirdparty/src/arrow-apache-arrow-0.15.1/cpp/release/zstd_ep-install/lib64
二 启动
1 启动FE
cd output/fe
mkdir doris-meta
bin/start_fe.sh --daemon
日志在log目录下
注意:
- 默认8030端口可能与yarn的resourcemanager冲突
- 启动之后检查fe绑定端口的ip是否正确,如果绑定ip错误(比如安装docker之后取到docker的ip),会导致be无法连接fe,需要配置fe.conf中的priority_networks,配置为正确的网段
priority_networks=192.168.1.0/24
2 启动BE
cd output/be
mkdir /path/to/storage
vim conf/be.conf
bin/start_be.sh --daemon
日志在log目录下
注意:
- 默认8040端口可能与yarn的nodemanager冲突
- 如果be启动失败,一般可能有两个原因:一个是端口被占用,一个是limit,根据日志排查,比如报错
Doris Be http service did not start correctly. exiting.
则是因为端口占用导致web启动失败
修改limit
临时修改
limit -n 65535
永久修改
vim /etc/security/limit.conf
* hard nofile 65535
* soft nofile 65535
3 启动Broker
cd fs_brokers/apache_hdfs_broker/output/apache_hdfs_broker
bin/start_broker.sh --daemon
三 使用
1 FE web访问
http://fe_server:8030
注意:
- 端口为http_port
- 默认账号root,密码空
2 FE 命令访问
mysql -P9030 -uroot
show proc '/frontends';
注意:
- 端口为query_port
3 添加或删除FE
alter system add follower '$host:$port';
show proc '/frontends';
4 添加或删除BE
alter system add backend '$host:$port';
alter system dropp backend '$host:$port';
show proc '/backends';
注意:
- port默认为9050,即heartbeat_service_port
5 添加或删除Broker
alter system add broker $broker_name '$host:$port';
show proc '/brokers';
注意:
- port默认为8000,即broker ipc_port
四 数据导入
hive数据导入
create database test;
CREATE TABLE test.test_user_doris
(
`id` varchar(128) ,
`name` varchar(128) ,
`country` varchar(128) ,
`province` varchar(128) ,
`city` varchar(128) ,
`order_count` int SUM
)
AGGREGATE KEY(id, name, country, province, city)
DISTRIBUTED BY HASH(id) BUCKETS 10
PROPERTIES("replication_num" = "3");
LOAD LABEL load_test_user_doris
(
DATA INFILE("hdfs://nameservice1/user/hive/warehouse/test.db/test_user/*")
INTO TABLE `test_user_doris`
FORMAT as "parquet"
)
WITH BROKER $broker_name
(
"dfs.nameservices" = "nameservice1",
"dfs.ha.namenodes.nameservice1" = "namenode1,namenode2",
"dfs.namenode.rpc-address.nameservice1.namenode1" = "nn1:8020",
"dfs.namenode.rpc-address.nameservice1.namenode2" = "nn2:8020"
)
PROPERTIES
(
"timeout"="36000",
"max_filter_ratio"="0.1"
);
show load;
五 参数配置
show variables;
同mysql
参考:
相关文章
- 【目标检测】MMDetection的安装与基础使用
- centos7 yum安装MongoDB[通俗易懂]
- A_A010_01 常用串口调试助手安装使用
- 06_Linux基础-NGINX和浏览器、网页的关系-云服务器ssh登陆-安装NGINX-上传网页-压缩命令-xz-gzip-bzip2-zip-tar-配置N
- AMD CPU电脑安装Android Studio详细步骤以及无法成功install HAXM的解决办法「建议收藏」
- k8s安装nginx
- Eclipse基础安装与简单使用
- Multipath多路径管理基础介绍与安装配置使用
- MsSQL2008R2安装基础流程笔记
- “站长,linux安装个软件咋那么费劲,有什么适合负基础人群的方法么?”
- Python基础-0 Python简介及PyCharm安装
- 【3】Pycharm超详细基础设置,autopep8 安装规范化程序,每个小trick都可以快速提升变成效率,超级实用!
- HarmonyOS实战[一]——原理概念介绍安装:基础篇
- Adobe Acrobat Pro DC 2018软件资源及安装教程-最牛逼的PDF编辑器
- PDF阅读器 PDF Acrobat Reader DC 2020中文版软件安装包免费下载以及安装教程
- CAD 2020软件安装包+安装教程CAD软件全版本下载
- MySql基础-笔记1 -Windows下安装配置图文教程
- 【Kotlin】Kotlin 语言基础 ( 环境安装 | 变量 | 简单使用 | 函数 | 异常 | 递归 )
- Ubuntu20.04基础:桌面安装及root权限开通和ssh安装讲解
- MacOS安装SVN简易指南(macos安装svn)
- 玩转Linux: 编译包快速安装指南(linux编译包)
- MinGW下载和安装教程
- 在CentOS 6.0 上安装LAMP服务器( Apache2 + PHP5 + MySQL)
- 系统Linux安装桌面系统:一步一步来(linux怎么装桌面)
- Linux 基础命令快速安装指南(linux基础命令安装)
- 修改MySQL安装后密码修改指南(mysql安装后密码)
- Linux系统服务器安装:一步一步构建稳定基础(linux系统服务器安装)
- Linux基础安装:开启稳定运行之旅(linux基本安装)
- 「零基础入门」Linux下安装QT教程,助力程序员成为高手!(linux下安装qt教程)
- 简单易懂:如何安装 Linux Bash?(linuxbash安装)
- Systemctl 安装及基础操作
- 服务器安装MSSQL:构建技术基础之路(服务器安装mssql)
- Linux安装根用户:从基础到高级(linux安装root)
- M1上快速安装Oracle数据库的指南(m1如何安装oracle)
- IIS7.5UrlScan3.1应用防火墙安装配置方法
- SUSELinux下源码编译方式安装MySQL5.6过程分享