Java calls Python Spark program to get stuck: How to solve the problem of Runtime.getRuntime().exec() blocking?

Analysis and solution of python code stuck in java call

In the process of calling python code using java, you often encounter some difficult problems, such as the program being stuck and unable to continue executing. This article will analyze a specific case and provide corresponding solutions.

Problem description: The developer uses Java's runtime.getruntime().exec() method to execute python scripts, and the python script uses spark for data processing. On the Java side, the output of the python script is obtained through the process object, but after the python script is executed to the sorted_word_count.take(20), the java side program is stuck and cannot continue execution.

The python script code is as follows:

 spark = sparksession.builder.appname("read from java backend").master("local[*]").getorcreate();

# Get the passed parameter comment = sys.argv[1]

# Convert json string to python object comment = json.loads(comment)

# Convert comment list to rdd
comment_rdd = spark.sparkcontext.parallelize(comment)

# Convert rdd to dataframe
df = spark.createdataframe(comment_rdd.map(lambda x: row(**x)))

# Load the stop word library stop_words = spark.sparkcontext.textfile("c:/users/10421/downloads/baidu_stopwords.txt").collect()

# ... (Some codes are omitted here) ...

# Calculate the number of occurrences of each word word_count = df.rdd.map(lambda x: (x.word, 1)).reducebykey(lambda x, y: xy)
sorted_word_count = word_count.sortby(lambda x: x[1], ascending=false)
top_20_words = sorted_word_count.take(20)
column = 0
for row in top_20_words:
    print(row[column])

The java code snippet is as follows:

 process process = runtime.getruntime().exec(args1);

// Get the program execution result inputstream inputstream = process.getinputstream();
bufferedreader reader = new bufferedreader(new inputstreamreader(inputstream,"gb2312"));
// ... (Some codes are omitted here) ...

Problem analysis: After testing, it was found that the reason why the Java program was stuck is the execution of the code sorted_word_count.take(20) in the python script. This part of the code will block until spark processing completes and returns the result. Since process.getinputstream() is blocking, if the output of the python program is not output to the standard output stream in time, the java program will wait for it, resulting in a stuck.

Solution: The problem is most likely in character encoding. The original code uses gb2312 encoding to read the output of python, which may be inconsistent with the output encoding of the python script, causing data read blockage. Modifying the java code and using utf-8 encoding to read the output of python can solve this problem.

Modified java code:

 BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, "UTF-8"));
BufferedReader reader2 = new BufferedReader(new InputStreamReader(errorStream, "UTF-8"));

By modifying the encoding of the read input stream and the error stream in the java code to utf-8, the problem of java program stuck can be solved. It should be noted that python scripts also need to make sure their output is encoded using utf-8. If the problem persists, you need to further check the execution efficiency of the spark job and whether there are other potential blocking operations in the python script.

The above is the detailed content of Java calls Python Spark program to get stuck: How to solve the problem of Runtime.getRuntime().exec() blocking?. For more information, please follow other related articles on the PHP Chinese website!