cannot find python3.7 when runing spark using spark-submit?

There is a problem that has bothered me for a few days. It occurs when I am running Spark application using spark-submit, it shows like this below:

09:14:36 Exception in thread "main" java.io.IOException: Cannot run program "python3.7": error=2, No such file or directory
09:14:36 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
09:14:36 at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:100)
09:14:36 at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
09:14:36 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
09:14:36 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
09:14:36 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
09:14:36 at java.lang.reflect.Method.invoke(Method.java:498)
09:14:36 at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
09:14:36 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
09:14:36 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
09:14:36 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
09:14:36 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
09:14:36 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
09:14:36 Caused by: java.io.IOException: error=2, No such file or directory
09:14:36 at java.lang.UNIXProcess.forkAndExec(Native Method)
09:14:36 at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
09:14:36 at java.lang.ProcessImpl.start(ProcessImpl.java:134)
09:14:36 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
09:14:36 ... 12 more


I have configured all the environment Variables both in ~/.bashrc and /etc/profile. And I am absolutely sure that all the environment variables are correct. And I can run python3.7 or PySpark in my Ubuntu.

~/.bashrc

/etc/profile

python3.7

export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export PATH=$PATH:$JAVA_HOME
export PATH=$PATH:/mwdata/python3.7/bin
export SPARK_HOME=/srv/spark
export PATH=$PATH:$SPARK_HOME/bin
export PYSPARK_PYTHON=python3.7
export PYSPARK_DRIVER_PYTHON=python3.7


So why doesn't it find python3.7?

python3.7

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.