spark on k8s准确完整的测试步骤:提交Pi计算程序至k8s集群
2023-09-14 09:01:51 时间
部署流程:
-
下载spark环境包https://www.apache.org/dyn/closer.lua/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
-
切换目录至根目录:cd spark-2.4.4-bin-hadoop2.7
-
builddocker应用程序镜像及push至repo:
docker build -t xxx/spark:2.4.4 -f kubernetes/dockerfiles/spark/Dockerfile . && docker push xxx/spark:2.4.4
-
创建aks serviceaccount:
kubectl create serviceaccount spark
-
创建clusterrolebinding:
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
-
本地提交应用程序(经验证:jar包路径及各项配置无误):
bin/spark-submit \
--master k8s://172.22.3.107:443 \
--deploy-mode cluster \
--conf spark.kubernetes.namespace=default \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=5 \
--conf spark.kubernetes.container.image=xxx/spark:2.4.0 \
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar
- docker提交spark程序(不推荐):
docker run -it --rm .kube/config:/root/.kube/config linclaus/spark-submit /opt/spark/bin/spark-submit --master k8s://https://data-extra-data-extraction-5abb9d-e807019a.hcp.chinanorth2.cx.prod.service.azk8s.cn:443 --deploy-mode cluster --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.namespace=default --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=1 --conf spark.kubernetes.container.image=linclaus/spark:2.4.4 local:///opt/spark/examples/jars/spark-examples_2.11-2.4.4.jar curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.16.0/bin/linux/amd64/kubectl
https://mirrors.tuna.tsinghua.edu.cn/apache/spark/
https://www.apache.org/dyn/closer.lua/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-linux
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.16.0/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
curl -L https://aka.ms/InstallAzureCli | bash
参考链接:
Spark On K8S
相关文章
- 从零开始入门 K8s | 应用存储和持久化数据卷:存储快照与拓扑调度
- 从零开始入门 K8s | 应用配置管理
- Centos7 k8s 基础单元pod
- k8s节点亲和性声明语法:键值运算符说明
- Spark on K8S及history服务部署
- k8s service:ClusterIP、NodePort、LoadBalancer、ExternalName
- 【云原生】Hadoop on k8s 环境部署
- 【云原生】ElasticSearch + Kibana on K8S 详解与实战操作
- ASP.NET Core on K8S学习初探(3)部署API到K8S
- 【云原生 • Kubernetes】k8s功能特性、k8s集群架构介绍
- 【云原生】Hive on k8s 环境部署
- 【云原生】Redis on k8s 编排部署讲解与实战操作
- K8s 应用的网络可观测性: Cilium VS DeepFlow
- 【K8S系列】第七讲:有状态服务 VS 无状态服务
- 【云原生】Hadoop on k8s 环境部署
- 【云原生】Spark on k8s 讲解与实战操作
- 【云原生】Flink on k8s 讲解与实战操作
- 【云原生】Azkaban on k8s 讲解与实战操作
- 【云原生】Presto/Trino on k8s 环境部署
- 在k8s集群中安装KubeSphere