Skip to content

Instantly share code, notes, and snippets.

@dbrambilla

dbrambilla/get_job_status.sh

Forked from arturmkrtchyan/get_job_status.sh
Created October 9, 2017 16:15
Show Gist options
  • Save dbrambilla/1ed7e8331a8461c5a0d38372667256d7 to your computer and use it in GitHub Desktop.
Save dbrambilla/1ed7e8331a8461c5a0d38372667256d7 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. @arturmkrtchyan arturmkrtchyan revised this gist Oct 8, 2015. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions kill_job_result.json
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,7 @@
    {
    "action" : "KillSubmissionResponse",
    "message" : "Kill request for driver-20151008181020-0015 submitted",
    "message" : "Kill request for driver-20151008145126-0000 submitted",
    "serverSparkVersion" : "1.5.0",
    "submissionId" : "driver-20151008181020-0015",
    "submissionId" : "driver-20151008145126-0000",
    "success" : true
    }
  2. @arturmkrtchyan arturmkrtchyan revised this gist Oct 8, 2015. 1 changed file with 7 additions and 0 deletions.
    7 changes: 7 additions & 0 deletions kill_job_result.json
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,7 @@
    {
    "action" : "KillSubmissionResponse",
    "message" : "Kill request for driver-20151008181020-0015 submitted",
    "serverSparkVersion" : "1.5.0",
    "submissionId" : "driver-20151008181020-0015",
    "success" : true
    }
  3. @arturmkrtchyan arturmkrtchyan revised this gist Oct 8, 2015. 3 changed files with 1 addition and 0 deletions.
    0 get_status.sh → get_job_status.sh
    View file
    File renamed without changes.
    0 get_status_result.json → get_job_status_result.json
    View file
    File renamed without changes.
    1 change: 1 addition & 0 deletions kill_job.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1 @@
    curl -X POST http://spark-cluster-ip:6066/v1/submissions/kill/driver-20151008145126-0000
  4. @arturmkrtchyan arturmkrtchyan revised this gist Oct 8, 2015. 2 changed files with 10 additions and 0 deletions.
    1 change: 1 addition & 0 deletions get_status.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1 @@
    curl http://spark-cluster-ip:6066/v1/submissions/status/driver-20151008145126-0000
    9 changes: 9 additions & 0 deletions get_status_result.json
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,9 @@
    {
    "action" : "SubmissionStatusResponse",
    "driverState" : "FINISHED",
    "serverSparkVersion" : "1.5.0",
    "submissionId" : "driver-20151008145126-0000",
    "success" : true,
    "workerHostPort" : "192.168.3.153:46894",
    "workerId" : "worker-20151007093409-192.168.3.153-46894"
    }
  5. @arturmkrtchyan arturmkrtchyan revised this gist Oct 8, 2015. 2 changed files with 9 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions submit_job.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,4 @@
    curl -X POST http://spark-master-ip:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
    curl -X POST http://spark-cluster-ip:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
    "action" : "CreateSubmissionRequest",
    "appArgs" : [ "myAppArgument1" ],
    "appResource" : "file:/myfilepath/spark-job-1.0.jar",
    @@ -13,6 +13,6 @@ curl -X POST http://spark-master-ip:6066/v1/submissions/create --header "Content
    "spark.app.name" : "MyJob",
    "spark.eventLog.enabled": "true",
    "spark.submit.deployMode" : "cluster",
    "spark.master" : "spark://spark-master-ip:6066"
    "spark.master" : "spark://spark-cluster-ip:6066"
    }
    }'
    7 changes: 7 additions & 0 deletions submit_job_result.json
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,7 @@
    {
    "action" : "CreateSubmissionResponse",
    "message" : "Driver successfully submitted as driver-20151008145126-0000",
    "serverSparkVersion" : "1.5.0",
    "submissionId" : "driver-20151008145126-0000",
    "success" : true
    }
  6. @arturmkrtchyan arturmkrtchyan created this gist Oct 8, 2015.
    18 changes: 18 additions & 0 deletions submit_job.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,18 @@
    curl -X POST http://spark-master-ip:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
    "action" : "CreateSubmissionRequest",
    "appArgs" : [ "myAppArgument1" ],
    "appResource" : "file:/myfilepath/spark-job-1.0.jar",
    "clientSparkVersion" : "1.5.0",
    "environmentVariables" : {
    "SPARK_ENV_LOADED" : "1"
    },
    "mainClass" : "com.mycompany.MyJob",
    "sparkProperties" : {
    "spark.jars" : "file:/myfilepath/spark-job-1.0.jar",
    "spark.driver.supervise" : "false",
    "spark.app.name" : "MyJob",
    "spark.eventLog.enabled": "true",
    "spark.submit.deployMode" : "cluster",
    "spark.master" : "spark://spark-master-ip:6066"
    }
    }'