How to install Hadoop in linux

1: Install JDK

1. Execute the following command to download the JDK1.8 installation package.

wget --no-check-certificate https://repo.huaweicloud.com/java/jdk/8u151-b12/jdk-8u151-linux-x64.tar.gz

2. Execute the following command to decompress the downloaded JDK1.8 installation package.

tar -zxvf jdk-8u151-linux-x64.tar.gz

3. Move and rename the JDK package.

mv jdk1.8.0_151/ /usr/java8

4. Configure Java environment variables.

echo 'export JAVA_HOME=/usr/java8' >> /etc/profile
echo 'export PATH=$PATH:$JAVA_HOME/bin' >> /etc/profile
source /etc/profile

5. Check whether Java is successfully installed.

java -version

2: Install Hadoop

Note: To download the Hadoop installation package, you can choose Huawei source (the speed is medium, acceptable, the focus is on the full version), Tsinghua source (3.0.0 or above The version download speed is too slow and there are few versions), Beijing Foreign Studies University source (the download speed is very fast, but there are few versions) - I personally tested it

1. Execute the following command to download Hadoop installation Bag.

wget --no-check-certificate https://repo.huaweicloud.com/apache/hadoop/common/hadoop-3.1.3/hadoop-3.1.3.tar.gz

2. Execute the following command to decompress the Hadoop installation package to /opt/hadoop.

tar -zxvf hadoop-3.1.3.tar.gz -C /opt/
mv /opt/hadoop-3.1.3 /opt/hadoop

3. Execute the following command to configure Hadoop environment variables.

echo 'export HADOOP_HOME=/opt/hadoop/' >> /etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/bin' >> /etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/sbin' >> /etc/profile
source /etc/profile

4. Execute the following command to modify the configuration files yarn-env.sh and hadoop-env.sh.

echo "export JAVA_HOME=/usr/java8" >> /opt/hadoop/etc/hadoop/yarn-env.sh
echo "export JAVA_HOME=/usr/java8" >> /opt/hadoop/etc/hadoop/hadoop-env.sh

5. Execute the following command to test whether Hadoop is installed successfully.

hadoop version

If version information is returned, the installation is successful.

3: Configure Hadoop

1. Modify the Hadoop configuration file core-site.xml.

a. Execute the following command to enter the editing page.

vim /opt/hadoop/etc/hadoop/core-site.xml

b. Enter i to enter edit mode. c. Insert the following content into the <configuration></configuration> node.

 <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/opt/hadoop/tmp</value>
        <description>location to store temporary files</description>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>

d. Press the Esc key to exit the editing mode, enter: wq to save and exit.

2. Modify the Hadoop configuration file hdfs-site.xml.

a. Execute the following command to enter the editing page.

vim /opt/hadoop/etc/hadoop/hdfs-site.xml

b. Enter i to enter edit mode. c. Insert the following content into the <configuration></configuration> node.

 <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/opt/hadoop/tmp/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/opt/hadoop/tmp/dfs/data</value>
    </property>

d. Press the Esc key to exit the editing mode, enter: wq to save and exit.

Four: Configure SSH password-free login

1. Execute the following command to create the public key and private key.

ssh-keygen -t rsa

2. Execute the following command to add the public key to the authorized_keys file.

cd ~
cd .ssh
cat id_rsa.pub >> authorized_keys

If an error is reported, perform the following operations and then re-execute the above two commands; if no error is reported, go directly to step five:

Enter the following command in the environment variable Add the following configuration

vi /etc/profile

Then add the following content to it

export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

Enter the following command to make the changes take effect

source /etc/profile

Five: Start Hadoop

1.Execute the following command to initialize the namenode.

hadoop namenode -format

2.Execute the following commands in sequence to start Hadoop.

start-dfs.sh

If Y/N is selected, select Y; otherwise press Enter directly

start-yarn.sh

3.After successful startup, execute the following command , to view the processes that have been successfully started.

jps

Normally there will be 6 processes;

4.Open the browser to visit http://:8088 and http://:50070. If the following interface is displayed, it means that the Hadoop pseudo-distributed environment is completed.

The above is the detailed content of How to install Hadoop in linux. For more information, please follow other related articles on the PHP Chinese website!