spark_1.x_submit_examples

------------------------------------------------------------------------------------------------ SPARK SUBMIT EXAMPLES ------------------------------------------------------------------------------------------------ Build Commands: -------------------------------------------------------------- cd /home/orienit/spark/workspace/KalyanSparkProjectMaven mvn clean eclipse:eclipse    (to prepare eclipse project) mvn package (to prepare jar file) cd /home/orienit/spark/workspace/KalyanSparkProjectSbt sbt clean eclipse    (to prepare eclipse project) sbt package (to prepare jar file) -------------------------------------------------------------- $SPARK_HOME/bin/run-example SparkPi 10 $SPARK_HOME/bin/run-example JavaWordCount file:///home/orienit/spark/input/demoinput $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10 $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.JavaWordCount --master local $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar file:///home/orienit/spark/input/demoinput $SPARK_HOME/bin/spark-submit $SPARK_HOME/examples/src/main/python/pi.py $SPARK_HOME/bin/spark-submit $SPARK_HOME/examples/src/main/python/wordcount.py file:///home/orienit/work/input/demoinput $SPARK_HOME/bin/spark-submit $SPARK_HOME/examples/src/main/r/dataframe.R Execute in Local Mode: =============================== $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10 $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local[4] $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10 $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local[*] $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10 Start Command for Spark: ----------------------------- $SPARK_HOME/sbin/start-all.sh Start Command for Spark: ----------------------------- $SPARK_HOME/sbin/stop-all.sh Execute in Cluster Mode: =============================== $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://kalyan:7077 $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10 $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://kalyan:7077 --deploy-mode client $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10 $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://kalyan:7077 --deploy-mode cluster $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10 Python example: =================== $SPARK_HOME/bin/spark-submit --master spark://kalyan:7077 $SPARK_HOME/examples/src/main/python/pi.py 10 Yarn example: ===================== export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10 $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10 Note: 1. --master yarn-client (or)    --master yarn --deploy-mode client 2. --master yarn-cluster (or)    --master yarn --deploy-mode cluster Own examples: ====================== $SPARK_HOME/bin/spark-submit --class --master $SPARK_HOME/bin/spark-submit --class com.orienit.spark.scala.training.WordCount --master local /home/orienit/spark/workspace-july-2016/KalyanSparkProjectSbt/target/scala-2.11/kalyansparkprojectsbt_2.11-1.0.jar file:///home/orienit/spark/input/demoinput file:///home/orienit/spark/output/demooutput $SPARK_HOME/bin/spark-submit --class com.orienit.spark.training.scala.WordCount --master local /home/orienit/spark/workspace-jan-16/SparkSbtProject1/target/scala-2.11/spark-project_2.11-1.0.jar file:///home/orienit/spark/input/demoinput file:///home/orienit/spark/output/demooutput-custom $SPARK_HOME/bin/spark-submit --class com.orienit.spark.training.WordCount --master local /home/orienit/spark/workspace/KalyanSparkProjectMaven/target/KalyanSparkProjectMaven-0.0.1-SNAPSHOT.jar file:///home/orienit/spark/input/demoinput $SPARK_HOME/bin/spark-submit --class com.orienit.spark.training.WordCount --master local /home/orienit/spark/workspace/KalyanSparkProjectSbt/target/scala-2.11/kalyansparkprojectsbt_2.11-1.0.jar file:///home/orienit/spark/input/demoinput

nikhil nikhil
Dec 3, 2024 - 17:18
0 1
------------------------------------------------------------------------------------------------
SPARK SUBMIT EXAMPLES
------------------------------------------------------------------------------------------------

Build Commands:
--------------------------------------------------------------

cd /home/orienit/spark/workspace/KalyanSparkProjectMaven
mvn clean eclipse:eclipse    (to prepare eclipse project)
mvn package (to prepare jar file)

cd /home/orienit/spark/workspace/KalyanSparkProjectSbt
sbt clean eclipse    (to prepare eclipse project)
sbt package (to prepare jar file)


--------------------------------------------------------------


$SPARK_HOME/bin/run-example SparkPi 10
$SPARK_HOME/bin/run-example JavaWordCount file:///home/orienit/spark/input/demoinput


$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.JavaWordCount --master local $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar file:///home/orienit/spark/input/demoinput

$SPARK_HOME/bin/spark-submit $SPARK_HOME/examples/src/main/python/pi.py

$SPARK_HOME/bin/spark-submit $SPARK_HOME/examples/src/main/python/wordcount.py file:///home/orienit/work/input/demoinput

$SPARK_HOME/bin/spark-submit $SPARK_HOME/examples/src/main/r/dataframe.R


Execute in Local Mode:
===============================
$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local[4] $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master local[*] $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10


Start Command for Spark:
-----------------------------
$SPARK_HOME/sbin/start-all.sh


Start Command for Spark:
-----------------------------
$SPARK_HOME/sbin/stop-all.sh



Execute in Cluster Mode:
===============================
$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://kalyan:7077 $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://kalyan:7077 --deploy-mode client $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://kalyan:7077 --deploy-mode cluster $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10



Python example:
===================
$SPARK_HOME/bin/spark-submit --master spark://kalyan:7077 $SPARK_HOME/examples/src/main/python/pi.py 10



Yarn example:
=====================
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster $SPARK_HOME/lib/spark-examples-1.6.0-hadoop2.6.0.jar 10


Note:
1. --master yarn-client
(or)
   --master yarn --deploy-mode client

2. --master yarn-cluster
(or)
   --master yarn --deploy-mode cluster





Own examples:
======================
$SPARK_HOME/bin/spark-submit --class
--master



$SPARK_HOME/bin/spark-submit --class com.orienit.spark.scala.training.WordCount --master local /home/orienit/spark/workspace-july-2016/KalyanSparkProjectSbt/target/scala-2.11/kalyansparkprojectsbt_2.11-1.0.jar file:///home/orienit/spark/input/demoinput file:///home/orienit/spark/output/demooutput


$SPARK_HOME/bin/spark-submit --class com.orienit.spark.training.scala.WordCount --master local /home/orienit/spark/workspace-jan-16/SparkSbtProject1/target/scala-2.11/spark-project_2.11-1.0.jar file:///home/orienit/spark/input/demoinput file:///home/orienit/spark/output/demooutput-custom


$SPARK_HOME/bin/spark-submit --class com.orienit.spark.training.WordCount --master local /home/orienit/spark/workspace/KalyanSparkProjectMaven/target/KalyanSparkProjectMaven-0.0.1-SNAPSHOT.jar file:///home/orienit/spark/input/demoinput

$SPARK_HOME/bin/spark-submit --class com.orienit.spark.training.WordCount --master local /home/orienit/spark/workspace/KalyanSparkProjectSbt/target/scala-2.11/kalyansparkprojectsbt_2.11-1.0.jar file:///home/orienit/spark/input/demoinput













Read More