提交程式

curl -X POST http://spark-cluster-ip:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
  "action" : "CreateSubmissionRequest",
  "appArgs" : [ "args1, args2,..." ], 
  "appResource" : "file:/myfilepath/spark-job-1.0.jar", 
  "clientSparkVersion" : "2.1.0",
  "environmentVariables"
 
 : {
    "SPARK_ENV_LOADED" : "1"
  },
  "mainClass" : "com.mycompany.MyJob",
  "sparkProperties" : {
    "spark.jars" : "file:/myfilepath/spark-job-1.0.jar",
    "spark.driver.supervise" : "false",
    "spark.app.name" : "MyJob",
    "spark.eventLog.enabled": "true",
    "spark.submit.deployMode" : "cluster"
 
,
    "spark.master" : "spark://spark-cluster-ip:6066"
  }
}'

說明：

• spark-cluster-ip：spark master地址。預設的rest服務埠是6066，如果被佔用會依次查詢6067,6068…

• “action” : “CreateSubmissionRequest”：請求的內容是提交程式

• “appArgs” : [ “args1, args2,…” ]：我們的程式jar包所需要的引數，如kafka topic，使用的模型等

  • 注意1：如果程式沒有需要的引數，這裡寫”appArgs”:[]，不能不寫，否則會把appResource後面的一條解析為appArgs引發未知錯誤

• “appResource” : “file:/myfilepath/spark-job-1.0.jar”：程式jar包的路徑

• “environmentVariables” : {“SPARK_ENV_LOADED” : “1”}：是否載入Spark環境變數

  • 注意2：此項必須要寫，否則會報NullPointException

• “clientSparkVersion” : “2.1.0”：spark的版本

• “mainClass” : “com.mycompany.MyJob”：程式的主類

• “sparkProperties” : {…}：spark的引數配置

響應結果：

{
  "action" : "CreateSubmissionResponse",
  "message" : "Driver successfully submitted as driver-20151008145126-0000",
  "serverSparkVersion" : "2.1.0",
  "submissionId" : "driver-20151008145126-0000",
  "success" : true
}

每項內容都很好理解，其中最重要的是submissionId和success，前者用於檢視和kill掉正在執行的spark程式，後者則可用來判斷當前程式是否提交成功。

獲取已提交程式的執行狀態

curl http://spark-cluster-ip:6066/v1/submissions/status/driver-20151008145126-0000

“driver-20151008145126-0000”則為上面提交程式後返回的submissionId

響應結果：

{
  "action" : "SubmissionStatusResponse",
  "driverState" : "FINISHED",
  "serverSparkVersion" : "2.1.0",
  "submissionId" : "driver-20151008145126-0000",
  "success" : true,
  "workerHostPort" : "192.168.3.153:46894",
  "workerId" : "worker-20151007093409-192.168.3.153-46894"
}

這裡driverState包括：ERROR(因錯誤沒有提交成功，會顯示錯誤資訊), SUBMITTED(已提交但未開始執行), RUNNIG(正在執行), FAILED(執行失敗，會丟擲異常), FINISHED(執行成功)

結束已提交的程式

curl -X POST http://spark-cluster-ip:6066/v1/submissions/kill/driver-20151008145126-0000

響應結果：

{
  "action" : "KillSubmissionResponse",
  "message" : "Kill request for driver-20151008145126-0000 submitted",
  "serverSparkVersion" : "2.1.0",
  "submissionId" : "driver-20151008145126-0000",
  "success" : true
}

可能遇到的問題

java.nio.charset.MalformedInputException: Input length = 1

原因：傳送請求的客戶端編碼與spark master端機器編碼不一致