Access to SparkUI based on Nginx reverse proxy

Access to SparkUI based on Nginx reverse proxy

Scenarios and solutions

In the case of spark cluster deployment, only the Master has a public IP and the Worker machine does not have a public IP. At this time, if you want to access the Run machine, run on the Master machine SparkUI, and when you need to view the logs on the Worker machine, there will be a problem of inaccessibility. One solution at this time is to use Nginx+SSH Tunnel port redirection to direct all requests from different ports on different machines to port 80 of the Master machine, and then forward them through Nginx as a reverse proxy.

Step breakdown

1. Deploy a Spark cluster and run SparkUI on the Master.

2. Deploy Nginx on the Master machine, and modify the $NGINX_HOME/conf/nginx.conf file to add the reverse proxy logic of the Work and Master machines, for example:

   <code>worker_processes  1;
   error_log  logs/error.log;
   error_log  logs/error.log  notice;
   error_log  logs/error.log  info;
   pid        logs/nginx.pid;
   
   events {
       worker_connections  1024;
   }
   
   http {
       include         mime.types;
       default_type    pplication/octet-stream;
       sendfile        off;
       keepalive_timeout  20;
       gzip  on;
       proxy_intercept_errors  off;
   
       upstream server_spark_master {
   
           server 127.0.0.1:8080;
   
       }
       upstream server_jyvhyguguiu-slave-0 {
   
           server 10.110.1.1:8081;
   
       }
       upstream server_jyvhyguguiu-slave-1 {
   
           server 10.110.1.2:8081;
   
       }
       server {
           listen 80;
           server_name spark_master;
           location / {
                   proxy_pass http://server_spark_master;
           }
   
       }
       server {
           listen 80;
           server_name jyvhyguguiu-slave-0;
           location / {
                   proxy_pass http://server_jyvhyguguiu-slave-0;
           }
       }
       server {
           listen 80;
           server_name jyvhyguguiu-slave-1;
           location / {
                   proxy_pass http://server_jyvhyguguiu-slave-1;
           }
       }       
   }
   </code>

Adopt the principle of nginx's machine name mapping, to Be a reverse proxy.

3. Bind the Master machine and Worker machine to 127.0.0.1 by modifying the hosts file.

4. Use the port forwarding principle supported by SSH to forward all the ports used by SparkUI to the remote monitoring Nginx service 80 port. For example, SparkUI will use the 8081 port of the Worker machine, the 8080 port of the Master machine, and the 80 port of the Master machine. Then all three local ports (because the Master and Worker machines are bound to the lo network card) will be forwarded to the remote 80 port. port. The command is as follows:

<code>ssh -N -f -L 8081:127.0.0.1:80 username@ip
ssh -N -f -L 8080:127.0.0.1:80 username@ip
ssh -N -f -L 80:127.0.0.1:80 username@ip
</code>

5. At this time, access the 127.0.0.1:8080 port to open the SparkUI of the remote machine, and you can access the logs of any Worker machine.

PS: In fact, SSH port forwarding can also use the Master machine as a proxy to forward the request to the Worker machine. The network path is the same as Nginx's reverse proxy, but SSH port forwarding can only achieve Layer 4, which is the transport layer. Forwarding cannot achieve layer 7 forwarding like Nginx (can identify the machine name and forward based on the machine name), so just through SSH port forwarding, the same effect cannot be achieved.

The above introduces the access to SparkUI based on Nginx reverse proxy, including aspects of the content. I hope it will be helpful to friends who are interested in PHP tutorials.