search
HomeDatabaseMysql Tutorialhadoop2 单机搭建

hadoop2 单机搭建

Jun 07, 2016 pm 03:08 PM
Standalonebuildenvironmentmatch

Hadoop2.2.0 环境配置说明(多节点配置在最后) 1.关闭防火墙 #chkconfigiptablesoff 2.检查状态 #chkconfig –list|grepiptables 全 off 即可 3.将 hadoop-2.2.0.tar.gz 文件复制到 /usr/local 目录下 4.解压 #tar –zxvfhadoop-2.2.0.tar.gz 5.改名 #mv Ha

Hadoop2.2.0环境配置说明(多节点配置在最后)

 

1. 关闭防火墙# chkconfig iptables off

2. 检查状态#chkconfig –list|grep iptables  off即可

3. 将hadoop-2.2.0.tar.gz文件复制到/usr/local目录下

4. 解压# tar –zxvf hadoop-2.2.0.tar.gz

5. 改名 # mv Hadoop-2.2.0 hadoop2.2

6. 修改环境变量 # vim /etc/profile

7. 添加 export HADOOP_HOME=/usr/local/hadoop2.2

     export HADOOP_MAPRED_HOME=$HADOOP_HOME

     export HADOOP_COMMON_HOME=$HADOOP_HOME

     export HADOOP_HDFS_HOME=$HADOOP_HOME 

     export YARN_HOME=$HADOOP_HOME

     export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

PATH下添加:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

8. 重载# source /etc/profile

9. 目录切换到 # cd /usr/local/hadoop2.2/etc/Hadoop

10. 在如下文件中添加对应内容

11. hadoop-env.sh  27行修改为

export JAVA_HOME=/usr/local/jdk1.6

12. yarn-env.sh 23行修改为

export JAVA_HOME=/usr/local/jdk1.6

13. 将mapred-site.xml.template 复制为mapred-site.xml

# cp mapred-site.xml.template mapred-site.xml

14. mapred-site.xml中第20(configuration)添加

    mapreduce.framework.name 

    yarn

15. yarn-site.xml18(configuration)添加

yarn.resourcemanager.hostname

    localhost

    hostanem of RM

yarn.resourcemanager.resource-tracker.address

localhost:5274

host is the hostname of the resource manager and

    port is the port on which the NodeManagers contact the Resource Manager.

    yarn.resourcemanager.scheduler.address

    localhost:5273

    host is the hostname of the resourcemanager and port is the port

    on which the Applications in the cluster talk to the Resource Manager.

 

    yarn.resourcemanager.scheduler.class

org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler

    In case you do not want to use the default scheduler

    yarn.resourcemanager.address

    localhost:5271

    the host is the hostname of the ResourceManager and the port is the port on which the clients can talk to the Resource Manager. 

 

    yarn.nodemanager.local-dirs

    

    the local directories used by the nodemanager

   yarn.nodemanager.address

   localhost:5272

   the nodemanagers bind to this port

  

   yarn.nodemanager.resource.memory-mb

   10240

  the amount of memory on the NodeManager in GB

   yarn.nodemanager.remote-app-log-dir

    /app-logs

    directory on hdfs where the application logs are moved to 

 

    yarn.nodemanager.log-dirs

    /usr/log

    the directories used by Nodemanagers as log directories

   yarn.nodemanager.aux-services

    mapreduce_shuffle

    shuffle service that needs to be set for Map Reduce to run 

16. core-site.xml20(configuration)添加

     hadoop.tmp.dir

     /usr/local/hadoop2.2/tmp

 

     fs.defaultFS 

     hdfs://localhost:9000 

     true 

17. hdfs-site.xml20(configuration)添加

dfs.namenode.name.dir

file:///dfs/name

true

 dfs.datanode.data.dir

  file:///dfs/data

 true

 

   dfs.replication

   1

  dfs.permissions.enabled

  false

 

hadoop-env.sh里面# export JAVA_HOME=/usr/local/jdk

 

18. 设置ssh   

# ssh-keygen –t rsa 一路回车默认值

进行查看 (应有id_rsaid_rsa.pub一对密钥文件)

# cd ~/.ssh

# ls 

复制出公钥

# cp id_rsa.pub authorized_keys

查看# ls  应有三个文件了

确认过程

# ssh localhost (输入yes)

# exit

# ssh localhost

19. 格式化 

# hadoop namenode –format

20. 启动 

#start-dfs.sh

#start-yarn.sh

21. 查看 # jps 应有6

22. 用自带浏览器,能打开http://localhost:50070/ 和http://localhost:8088/cluster即可

(多机环境配置)

23. DNS配置(建议修改之后重启虚拟机)

24. 将主机名换为hadoop2

# vim /etc/sysconfig/network

2行,localhost.localdomain改为hadoop2

25. 修改地址映射

# vim /etc/hosts

第三行添加192.168.100.11 hadoop2 (地址不固定)

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
How does MySQL differ from SQLite?How does MySQL differ from SQLite?Apr 24, 2025 am 12:12 AM

The main difference between MySQL and SQLite is the design concept and usage scenarios: 1. MySQL is suitable for large applications and enterprise-level solutions, supporting high performance and high concurrency; 2. SQLite is suitable for mobile applications and desktop software, lightweight and easy to embed.

What are indexes in MySQL, and how do they improve performance?What are indexes in MySQL, and how do they improve performance?Apr 24, 2025 am 12:09 AM

Indexes in MySQL are an ordered structure of one or more columns in a database table, used to speed up data retrieval. 1) Indexes improve query speed by reducing the amount of scanned data. 2) B-Tree index uses a balanced tree structure, which is suitable for range query and sorting. 3) Use CREATEINDEX statements to create indexes, such as CREATEINDEXidx_customer_idONorders(customer_id). 4) Composite indexes can optimize multi-column queries, such as CREATEINDEXidx_customer_orderONorders(customer_id,order_date). 5) Use EXPLAIN to analyze query plans and avoid

Explain how to use transactions in MySQL to ensure data consistency.Explain how to use transactions in MySQL to ensure data consistency.Apr 24, 2025 am 12:09 AM

Using transactions in MySQL ensures data consistency. 1) Start the transaction through STARTTRANSACTION, and then execute SQL operations and submit it with COMMIT or ROLLBACK. 2) Use SAVEPOINT to set a save point to allow partial rollback. 3) Performance optimization suggestions include shortening transaction time, avoiding large-scale queries and using isolation levels reasonably.

In what scenarios might you choose PostgreSQL over MySQL?In what scenarios might you choose PostgreSQL over MySQL?Apr 24, 2025 am 12:07 AM

Scenarios where PostgreSQL is chosen instead of MySQL include: 1) complex queries and advanced SQL functions, 2) strict data integrity and ACID compliance, 3) advanced spatial functions are required, and 4) high performance is required when processing large data sets. PostgreSQL performs well in these aspects and is suitable for projects that require complex data processing and high data integrity.

How can you secure a MySQL database?How can you secure a MySQL database?Apr 24, 2025 am 12:04 AM

The security of MySQL database can be achieved through the following measures: 1. User permission management: Strictly control access rights through CREATEUSER and GRANT commands. 2. Encrypted transmission: Configure SSL/TLS to ensure data transmission security. 3. Database backup and recovery: Use mysqldump or mysqlpump to regularly backup data. 4. Advanced security policy: Use a firewall to restrict access and enable audit logging operations. 5. Performance optimization and best practices: Take into account both safety and performance through indexing and query optimization and regular maintenance.

What are some tools you can use to monitor MySQL performance?What are some tools you can use to monitor MySQL performance?Apr 23, 2025 am 12:21 AM

How to effectively monitor MySQL performance? Use tools such as mysqladmin, SHOWGLOBALSTATUS, PerconaMonitoring and Management (PMM), and MySQL EnterpriseMonitor. 1. Use mysqladmin to view the number of connections. 2. Use SHOWGLOBALSTATUS to view the query number. 3.PMM provides detailed performance data and graphical interface. 4.MySQLEnterpriseMonitor provides rich monitoring functions and alarm mechanisms.

How does MySQL differ from SQL Server?How does MySQL differ from SQL Server?Apr 23, 2025 am 12:20 AM

The difference between MySQL and SQLServer is: 1) MySQL is open source and suitable for web and embedded systems, 2) SQLServer is a commercial product of Microsoft and is suitable for enterprise-level applications. There are significant differences between the two in storage engine, performance optimization and application scenarios. When choosing, you need to consider project size and future scalability.

In what scenarios might you choose SQL Server over MySQL?In what scenarios might you choose SQL Server over MySQL?Apr 23, 2025 am 12:20 AM

In enterprise-level application scenarios that require high availability, advanced security and good integration, SQLServer should be chosen instead of MySQL. 1) SQLServer provides enterprise-level features such as high availability and advanced security. 2) It is closely integrated with Microsoft ecosystems such as VisualStudio and PowerBI. 3) SQLServer performs excellent in performance optimization and supports memory-optimized tables and column storage indexes.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

WebStorm Mac version

WebStorm Mac version

Useful JavaScript development tools

ZendStudio 13.5.1 Mac

ZendStudio 13.5.1 Mac

Powerful PHP integrated development environment

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)