search
HomeDatabaseMysql Tutorialhadoop 配置机架感知

hadoop 配置机架感知

Jun 07, 2016 pm 04:31 PM
hadoopPerceptionConfiguration

周海汉?2013.7.24 http://abloz.com 假如设备链接层次分3层,第一层交换机d1下面连多个交换机rk1,rk2,rk3,rk4,. 每个交换机对应一个机架。 d1(rk1(hs11,hs12,),rk2(hs21,hs22,), rk3(hs31,hs32,),rk4(hs41,hs42,),) 可以用程序或脚本完成由host到设备的映射

周海汉?2013.7.24

http://abloz.com

假如设备链接层次分3层,第一层交换机d1下面连多个交换机rk1,rk2,rk3,rk4,…. 每个交换机对应一个机架。

d1(rk1(hs11,hs12,…),rk2(hs21,hs22,…), rk3(hs31,hs32,…),rk4(hs41,hs42,…),…)

可以用程序或脚本完成由host到设备的映射。比如,用python,生成一个topology.py:

然后在core-site.xml中配置

topology.script.file.name
/home/hadoop/hadoop-1.1.2/conf/topology.py
The script name that should be invoked to resolve DNS names to
NetworkTopology names. Example: the script would take host.foo.bar as an
argument, and return /rack1 as the output.

python机架脚本:

[hadoop@hs11 conf]$ cat topology.py
#!/usr/bin/env python

”’
This script used by hadoop to determine network/rack topology. It
should be specified in hadoop-site.xml via topology.script.file.name
Property.
topology.script.file.name
/home/hadoop/hadoop-1.1.2/conf/topology.py

To generate dict:
for i in range(xx):
#print “\”hs%d\”:\”/rk%d/hs%d\”,”%(i,(i-1)/10,i)

print “\”hs%d\”:\”/rk%d\”,”%(i,(i-1)/10)

Andy 2013.7.23
”’

import sys
from string import join

DEFAULT_RACK = ‘/rk0′;

RACK_MAP = {
“hs11″:”/rk1″,
“hs12″:”/rk1″,
“hs13″:”/rk1″,
“hs14″:”/rk1″,
“hs15″:”/rk1″,
“hs16″:”/rk1″,
“hs17″:”/rk1″,
“hs18″:”/rk1″,
“hs19″:”/rk1″,
“hs20″:”/rk1″,
“hs21″:”/rk2″,
“hs22″:”/rk2″,
“hs23″:”/rk2″,
“hs24″:”/rk2″,
“hs25″:”/rk2″,
“hs26″:”/rk2″,
“hs27″:”/rk2″,
“hs28″:”/rk2″,
“hs29″:”/rk2″,
“hs30″:”/rk2″,
“hs31″:”/rk3″,
“hs32″:”/rk3″,
“hs33″:”/rk3″,
“hs34″:”/rk3″,
“hs35″:”/rk3″,
“hs36″:”/rk3″,
“hs37″:”/rk3″,
“hs38″:”/rk3″,
“hs39″:”/rk3″,
“hs40″:”/rk3″,
“hs41″:”/rk4″,
“hs42″:”/rk4″,
“hs43″:”/rk4″,
“hs44″:”/rk4″,
“hs45″:”/rk4″,
“hs46″:”/rk4″,

“10.10.20.11″:”/rk1″,
“10.10.20.12″:”/rk1″,
“10.10.20.13″:”/rk1″,
“10.10.20.14″:”/rk1″,
“10.10.20.15″:”/rk1″,
“10.10.20.16″:”/rk1″,
“10.10.20.17″:”/rk1″,
“10.10.20.18″:”/rk1″,
“10.10.20.19″:”/rk1″,
“10.10.20.20″:”/rk1″,
“10.10.20.21″:”/rk2″,
“10.10.20.22″:”/rk2″,
“10.10.20.23″:”/rk2″,
“10.10.20.24″:”/rk2″,
“10.10.20.25″:”/rk2″,
“10.10.20.26″:”/rk2″,
“10.10.20.27″:”/rk2″,
“10.10.20.28″:”/rk2″,
“10.10.20.29″:”/rk2″,
“10.10.20.30″:”/rk2″,
“10.10.20.31″:”/rk3″,
“10.10.20.32″:”/rk3″,
“10.10.20.33″:”/rk3″,
“10.10.20.34″:”/rk3″,
“10.10.20.35″:”/rk3″,
“10.10.20.36″:”/rk3″,
“10.10.20.37″:”/rk3″,
“10.10.20.38″:”/rk3″,
“10.10.20.39″:”/rk3″,
“10.10.20.40″:”/rk3″,
“10.10.20.41″:”/rk4″,
“10.10.20.42″:”/rk4″,
“10.10.20.43″:”/rk4″,
“10.10.20.44″:”/rk4″,
“10.10.20.45″:”/rk4″,
“10.10.20.46″:”/rk4″,


}

if len(sys.argv)==1:
print DEFAULT_RACK
else:
print join([RACK_MAP.get(i, DEFAULT_RACK) for i in sys.argv[1:]],” “)

原来这个程序我返回的是

“hs11″:”/rk1/hs11″,

结果执行mapreduce程序时报如下错误:

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there’s no reduce operator
Starting Job = job_201307241502_0003, Tracking URL = http://hs11:50030/jobdetails.jsp?jobid=job_201307241502_0003
Kill Command = /home/hadoop/hadoop-1.1.2/libexec/../bin/hadoop job? -kill job_201307241502_0003
Hadoop job information for Stage-1: number of mappers: 0; number of reducers: 0
2013-07-24 18:38:11,854 Stage-1 map = 100%,? reduce = 100%
Ended Job = job_201307241502_0003 with errors
Error during job, obtaining debugging information…
Job Tracking URL: http://hs11:50030/jobdetails.jsp?jobid=job_201307241502_0003
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0:? HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

通过http://hs11:50030/jobdetails.jsp?jobid=job_201307241502_0002?可以看到:

Job initialization failed:

java.lang.NullPointerException

at?org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2751)
at?org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:578)
at?org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:750)

at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3775)

at?org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:90)
at?java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at?java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

原来系统在配置机架敏感时,并不需要在脚本中返回设备ns或hostname,系统会自动添加。改为上面的topology.py后,系统执行正确。

相关博文:

  1. hadoop 打印配置变量
  2. hadoop 中的 ClassNotFoundException
  3. hadoop ubuntu集群安装
Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
How does MySQL handle transactions compared to other relational databases?How does MySQL handle transactions compared to other relational databases?Apr 29, 2025 am 12:37 AM

MySQLhandlestransactionseffectivelyusingtheInnoDBengine,supportingACIDpropertiessimilartoPostgreSQLandOracle.1)MySQLusesREPEATABLEREADasthedefaultisolationlevel,whichcanbeadjustedtoREADCOMMITTEDforhigh-trafficscenarios.2)Itoptimizesperformancewithabu

How does MySQL differ from PostgreSQL?How does MySQL differ from PostgreSQL?Apr 29, 2025 am 12:23 AM

MySQLisbetterforspeedandsimplicity,suitableforwebapplications;PostgreSQLexcelsincomplexdatascenarioswithrobustfeatures.MySQLisidealforquickprojectsandread-heavytasks,whilePostgreSQLispreferredforapplicationsrequiringstrictdataintegrityandadvancedSQLf

How does MySQL handle data replication?How does MySQL handle data replication?Apr 28, 2025 am 12:25 AM

MySQL processes data replication through three modes: asynchronous, semi-synchronous and group replication. 1) Asynchronous replication performance is high but data may be lost. 2) Semi-synchronous replication improves data security but increases latency. 3) Group replication supports multi-master replication and failover, suitable for high availability requirements.

How can you use the EXPLAIN statement to analyze query performance?How can you use the EXPLAIN statement to analyze query performance?Apr 28, 2025 am 12:24 AM

The EXPLAIN statement can be used to analyze and improve SQL query performance. 1. Execute the EXPLAIN statement to view the query plan. 2. Analyze the output results, pay attention to access type, index usage and JOIN order. 3. Create or adjust indexes based on the analysis results, optimize JOIN operations, and avoid full table scanning to improve query efficiency.

How do you back up and restore a MySQL database?How do you back up and restore a MySQL database?Apr 28, 2025 am 12:23 AM

Using mysqldump for logical backup and MySQLEnterpriseBackup for hot backup are effective ways to back up MySQL databases. 1. Use mysqldump to back up the database: mysqldump-uroot-pmydatabase>mydatabase_backup.sql. 2. Use MySQLEnterpriseBackup for hot backup: mysqlbackup--user=root-password=password--backup-dir=/path/to/backupbackup. When recovering, use the corresponding life

What are some common causes of slow queries in MySQL?What are some common causes of slow queries in MySQL?Apr 28, 2025 am 12:18 AM

The main reasons for slow MySQL query include missing or improper use of indexes, query complexity, excessive data volume and insufficient hardware resources. Optimization suggestions include: 1. Create appropriate indexes; 2. Optimize query statements; 3. Use table partitioning technology; 4. Appropriately upgrade hardware.

What are views in MySQL?What are views in MySQL?Apr 28, 2025 am 12:04 AM

MySQL view is a virtual table based on SQL query results and does not store data. 1) Views simplify complex queries, 2) Enhance data security, and 3) Maintain data consistency. Views are stored queries in databases that can be used like tables, but data is generated dynamically.

What are the differences in syntax between MySQL and other SQL dialects?What are the differences in syntax between MySQL and other SQL dialects?Apr 27, 2025 am 12:26 AM

MySQLdiffersfromotherSQLdialectsinsyntaxforLIMIT,auto-increment,stringcomparison,subqueries,andperformanceanalysis.1)MySQLusesLIMIT,whileSQLServerusesTOPandOracleusesROWNUM.2)MySQL'sAUTO_INCREMENTcontrastswithPostgreSQL'sSERIALandOracle'ssequenceandt

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

MantisBT

MantisBT

Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

ZendStudio 13.5.1 Mac

ZendStudio 13.5.1 Mac

Powerful PHP integrated development environment

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.