spark-submit提交方式测试Demo详解大数据

maven.compiler.source 1.7 /maven.compiler.source maven.compiler.target 1.7 /maven.compiler.target encoding UTF-8 /encoding spark.version 1.6.1 /spark.version /properties dependencies dependency groupId org.apache.spark /groupId artifactId spark-core_2.10 /artifactId version ${spark.version} /version /dependency dependency groupId redis.clients /groupId artifactId jedis /artifactId version 2.7.1 /version /dependency /dependencies build plugins plugin groupId org.apache.maven.plugins /groupId artifactId maven-compiler-plugin /artifactId configuration source 1.7 /source target 1.7 /target /configuration /plugin plugin groupId org.apache.maven.plugins /groupId artifactId maven-shade-plugin /artifactId version 2.4.3 /version executions execution phase package /phase goals goal shade /goal /goals configuration filters filter artifact *:* /artifact excludes exclude META-INF/*.SF /exclude exclude META-INF/*.DSA /exclude exclude META-INF/*.RSA /exclude /excludes /filter /filters /configuration /execution /executions /plugin /plugins /build

编写一个蒙特卡罗求PI的代码

import java.util.ArrayList; 

import java.util.List; 

import org.apache.spark.SparkConf; 

import org.apache.spark.api.java.JavaRDD; 

import org.apache.spark.api.java.JavaSparkContext; 

import org.apache.spark.api.java.function.Function; 

import org.apache.spark.api.java.function.Function2; 

import redis.clients.jedis.Jedis; 

 * Computes an approximation to pi 

 * Usage: JavaSparkPi [slices] 

public final class JavaSparkPi { 

 public static void main(String[] args) throws Exception { 

 SparkConf sparkConf = new SparkConf().setAppName("JavaSparkPi")/*.setMaster("local[2]")*/; 

 JavaSparkContext jsc = new JavaSparkContext(sparkConf); 

 Jedis jedis = new Jedis("192.168.49.151",19000); 

 int slices = (args.length == 1) ? Integer.parseInt(args[0]) : 2; 

 int n = 100000 * slices; 

 List Integer l = new ArrayList Integer (n); 

 for (int i = 0; i i++) { 

 l.add(i); 

 JavaRDD Integer dataSet = jsc.parallelize(l, slices); 

 int count = dataSet.map(new Function Integer, Integer () { 

 @Override 

 public Integer call(Integer integer) { 

 double x = Math.random() * 2 - 1; 

 double y = Math.random() * 2 - 1; 

 return (x * x + y * y 1) ? 1 : 0; 

 }).reduce(new Function2 Integer, Integer, Integer () { 

 @Override 

 public Integer call(Integer integer, Integer integer2) { 

 return integer + integer2; 

 }); 

 jedis.set("Pi", String.valueOf(4.0 * count / n)); 

 System.out.println("Pi is roughly " + 4.0 * count / n); 

 jsc.stop(); 

}

 

前提条件的setMaster( local[2] ) 没有在代码中hard code

本地模式测试情况：# Run application locally on 8 cores

spark-submit /
master local[8] /
class com.spark.test.JavaSparkPi /
executor-memory 4g /
executor-cores 4 /
/home/dinpay/test/Spark-SubmitTest.jar 100

运行结果在本地：运行在本地一起提交8个Task，不会在WebUI的8080端口上看见提交的任务

 

-

spark-submit /
master local[8] /
class com.spark.test.JavaSparkPi /
executor-memory 8G /
total-executor-cores 8 /
hdfs://192.168.46.163:9000/home/test/Spark-SubmitTest.jar 100

运行报错：java.lang.ClassNotFoundException: com.spark.test.JavaSparkPi

spark-submit /
master local[8] /
deploy-mode cluster /
supervise /
class com.spark.test.JavaSparkPi /
executor-memory 8G /
total-executor-cores 8 /
/home/dinpay/test/Spark-SubmitTest.jar 100

运行报错:Error: Cluster deploy mode is not compatible with master local

====================================================================

Standalone模式client模式 # Run on a Spark standalone cluster in client deploy mode

spark-submit /
master spark://hadoop-namenode-02:7077 /
class com.spark.test.JavaSparkPi /
executor-memory 8g /
tital-executor-cores 8 /
/home/dinpay/test/Spark-SubmitTest.jar 100

运行结果如下：

 

-
spark-submit /
master spark://hadoop-namenode-02:7077 /
class com.spark.test.JavaSparkPi /
executor-memory 4g /
executor-cores 4g /
hdfs://192.168.46.163:9000/home/test/Spark-SubmitTest.jar 100

运行报错:java.lang.ClassNotFoundException: com.spark.test.JavaSparkPi

 

=======================================================================

standalone模式下的cluster模式 # Run on a Spark standalone cluster in cluster deploy mode with supervise

spark-submit /
master spark://hadoop-namenode-02:7077 /
class com.spark.test.JavaSparkPi /
deploy-mode cluster /
supervise /
executor-memory 4g /
executor-cores 4 /
/home/dinpay/test/Spark-SubmitTest.jar 100

运行报错：java.io.FileNotFoundException: /home/dinpay/test/Spark-SubmitTest.jar (No such file or directory)

-

spark-submit /
master spark://hadoop-namenode-02:7077 /
class com.spark.test.JavaSparkPi /
deploy-mode cluster /
supervise /
driver-memory 4g /
driver-cores 4 /
executor-memory 2g /
total-executor-cores 4 /
hdfs://192.168.46.163:9000/home/test/Spark-SubmitTest.jar 100

运行结果如下：

 

=============================================

如果代码中写定了.setMaster( local[2] );
则提交的集群模式也会运行driver，但是不会有对应的application并行运行

spark-submit deploy-mode cluster /
master spark://hadoop-namenode-02:6066 /
class com.dinpay.bdp.rcp.service.Window12HzStat /
driver-memory 2g /
driver-cores 2 /
executor-memory 1g /
total-executor-cores 2 /
hdfs://192.168.46.163:9000/home/dinpay/RCP-HZ-TASK-0.0.1-SNAPSHOT.jar
如果代码中限定了.setMaster( local[2] );
则提交方式还是本地模式，会找一台worker进行本地化运行任务

 

原创文章，作者：ItWorker，如若转载，请注明出处：https://blog.ytso.com/9050.html

分布式文件系统，分布式数据库区块链并行处理（MPP）数据库，数据挖掘开源大数据平台数据中台数据分析数据开发数据治理数据湖数据采集