How to solve the problem that Flink cannot find Python task script when submitting PyFlink job to Yarn Application?

Solution to Python scripts not found when Flink submits PyFlink job to Yarn

When submitting PyFlink jobs to Yarn using Flink, if you encounter an error in which the Python script cannot be found, it is usually caused by a Python script path configuration error or a Python environment setting problem. This article analyzes and resolves this issue.

You submitted a PyFlink job using the following command:

 ./flink run-application -t yarn-application \
-dyarn.application.name=flinkcdctestpython\
-dyarn.provided.lib.dirs="hdfs://nameservice1/pyflink/flink-dist-181" \
-pyarch hdfs://nameservice1/pyflink/pyflink181.zip \
-pyclientexec pyflink181.zip/pyflink181/bin/python \
-pyexec pyflink181.zip/pyflink181/bin/python \
-py hdfs://nameservice1/pyflink/wc2.py

The error message is as follows:

 <code>2024-05-24 16:38:02,030 info org.apache.flink.client.python.pythondriver [] - pyflink181.zip/pyflink181/bin/python: can't open file 'hdfs://nameservice1/pyflink/wc2.py': [errno 2] no such file or directory</code>

This error indicates that Flink cannot find the specified Python script wc2.py However, the HDFS configuration is normal when submitting the Java job, which means there is no problem with the HDFS configuration itself.

The problems may lie in the following aspects:

1. Python script path: double check whether the hdfs://nameservice1/pyflink/wc2.py path is correct and whether the wc2.py file exists under this path. Verify using HDFS command:

    hdfs dfs -ls hdfs://nameservice1/pyflink/wc2.py

2. Python environment configuration: -pyclientexec and -pyexec parameters specify the Python execution environment. Make sure the Python environment in pyflink181.zip is configured correctly and has access to HDFS. It is recommended to point the parameters directly to the Python environment path on HDFS:

    -pyclientexec hdfs://nameservice1/pyflink/pyflink181.zip/pyflink181/bin/python
   -pyexec hdfs://nameservice1/pyflink/pyflink181.zip/pyflink181/bin/python

3. Permissions Issue: Make sure the Flink job has permission to access Python script files on HDFS. Check file permissions:

    hdfs dfs -ls -h hdfs://nameservice1/pyflink/wc2.py

4. Flink and PyFlink version compatibility: Confirm the Flink version is compatible with the PyFlink version. Version mismatch can cause problems.

Through the above steps, you should be able to find and resolve the issue that Flink cannot find the Python script when submitting PyFlink jobs. If the problem persists, check Flink and PyFlink's log files for more clues.

The above is the detailed content of How to solve the problem that Flink cannot find Python task script when submitting PyFlink job to Yarn Application?. For more information, please follow other related articles on the PHP Chinese website!