Why Do I Get a 403 Forbidden Error with My Java Google Search Program, But Not in My Web Browser?

403 Forbidden with Java but Not Web Browser: Solving the Google Search Issue

Despite the successful retrieval of search results in web browsers, encountering a 403 Forbidden error when attempting to fetch Google search data using a Java program can be puzzling. The root cause lies in the absence of user agent information in the Java request.

To resolve this issue, it is necessary to simulate the behavior of a web browser by setting the "User-Agent" header:

URLConnection connection = new URL("https://www.google.com/search?q=" + query).openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
connection.connect();

This header instructs the server to treat the request as originating from a web browser, which prevents the 403 error. It is worth noting that SSL handling is handled automatically.

However, retrieving the result amount using Java necessitates further steps. It is necessary to fetch the cookie and parse the redirect token link:

String cookie = connection.getHeaderField("Set-Cookie").split(";")[0];
Pattern pattern = Pattern.compile("content=\\"0;url=(.*?)\\"");
Matcher m = pattern.matcher(response);
if (m.find()) {
    String url = m.group(1);
    connection = new URL(url).openConnection();
    connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
    connection.setRequestProperty("Cookie", cookie);
    connection.connect();
    r = new BufferedReader(new InputStreamReader(connection.getInputStream(), Charset.forName("UTF-8")));
    sb = new StringBuilder();
    while ((line = r.readLine()) != null) {
        sb.append(line);
    }
    response = sb.toString();
    pattern = Pattern.compile("<div>

Executing the entire code block yields a result of 2930000000L, indicating the number of search results.

The above is the detailed content of Why Do I Get a 403 Forbidden Error with My Java Google Search Program, But Not in My Web Browser?. For more information, please follow other related articles on the PHP Chinese website!