最近要做一次关于yarn的分享,于是想搭建一个Hadoop环境。Hadoop 2.0较之前的Hadoop 0.1x变化比较大,折腾了好久了,终于把环境搞好了。我搭建了一个两节点的集群,只配置了一些必须的参数,让集群勉强跑起来。 1、core-site.xml configurationpropertynamef
最近要做一次关于yarn的分享,于是想搭建一个Hadoop环境。Hadoop 2.0较之前的Hadoop 0.1x变化比较大,折腾了好久了,终于把环境搞好了。我搭建了一个两节点的集群,只配置了一些必须的参数,让集群勉强跑起来。
1、core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://10.232.42.91:19000/</value> </property> </configuration>
2、mapred-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
3、yarn-site.xml
<configuration> <property> <name>yarn.resourcemanager.address</name> <value>hdfs://10.232.42.91:19001/</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>hdfs://10.232.42.91:19002/</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce.shuffle</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>10.232.42.91:8030</value> </property> </configuration>
把JAVA_HOME、HADOOP_HOME都设置到.bashrc里面去,然后运行sbin/start-all.sh。使用jps可以看到两个节点下运行的进程如下。
[master] jps 31318 ResourceManager 28981 DataNode 11580 JobHistoryServer 28858 NameNode 29155 SecondaryNameNode 31426 NodeManager 11016 Jps [slave] jps 12592 NodeManager 11711 DataNode 17699 Jps
上面这个JobHistoryServer需要单独启动,通过它可以看到每个application的详细日志。启动命令如下。
sbin/mr-jobhistory-daemon.sh start historyserver
打开http://10.232.42.91:8088/cluster/cluster这个地址可以看到cluster的介绍信息。这里再也看不到slot相关的数据了。
万事俱备。放点文本数据到hdfs://10.232.42.91:19000/input这个目录下,运行wordcount看看效果。
$ cd hadoop/share/hadoop/mapreduce $ hadoop jar hadoop-mapreduce-examples-2.0.3-alpha.jar wordcount hdfs://10.232.42.91:19000/input hdfs://10.232.42.91:19000/output 13/03/07 21:08:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 13/03/07 21:08:26 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/03/07 21:08:26 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. 13/03/07 21:08:26 INFO input.FileInputFormat: Total input paths to process : 3 13/03/07 21:08:26 INFO mapreduce.JobSubmitter: number of splits:3 13/03/07 21:08:26 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar 13/03/07 21:08:26 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 13/03/07 21:08:26 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class 13/03/07 21:08:26 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 13/03/07 21:08:26 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name 13/03/07 21:08:26 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class 13/03/07 21:08:26 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 13/03/07 21:08:26 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 13/03/07 21:08:26 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 13/03/07 21:08:26 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 13/03/07 21:08:26 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 13/03/07 21:08:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1362658309553_0019 13/03/07 21:08:26 INFO client.YarnClientImpl: Submitted application application_1362658309553_0019 to ResourceManager at /10.232.42.91:19001 13/03/07 21:08:26 INFO mapreduce.Job: The url to track the job: http://search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0019/ 13/03/07 21:08:26 INFO mapreduce.Job: Running job: job_1362658309553_0019 13/03/07 21:08:33 INFO mapreduce.Job: Job job_1362658309553_0019 running in uber mode : false 13/03/07 21:08:33 INFO mapreduce.Job: map 0% reduce 0% 13/03/07 21:08:39 INFO mapreduce.Job: map 100% reduce 0% 13/03/07 21:08:44 INFO mapreduce.Job: map 100% reduce 100% 13/03/07 21:08:44 INFO mapreduce.Job: Job job_1362658309553_0019 completed successfully 13/03/07 21:08:44 INFO mapreduce.Job: Counters: 43 File System Counters FILE: Number of bytes read=12698 FILE: Number of bytes written=312593 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=16947 HDFS: Number of bytes written=8739 HDFS: Number of read operations=12 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=3 Launched reduce tasks=1 Rack-local map tasks=3 Total time spent by all maps in occupied slots (ms)=10750 Total time spent by all reduces in occupied slots (ms)=4221 Map-Reduce Framework Map input records=317 Map output records=2324 Map output bytes=24586 Map output materialized bytes=12710 Input split bytes=316 Combine input records=2324 Combine output records=885 Reduce input groups=828 Reduce shuffle bytes=12710 Reduce input records=885 Reduce output records=828 Spilled Records=1770 Shuffled Maps =3 Failed Shuffles=0 Merged Map outputs=3 GC time elapsed (ms)=376 CPU time spent (ms)=4480 Physical memory (bytes) snapshot=557428736 Virtual memory (bytes) snapshot=2105122816 Total committed heap usage (bytes)=254607360 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=16631 File Output Format Counters Bytes Written=8739
接下来玩玩yarn吧。Hadoop官方文档那篇WritingYarnApplications太让人蛋碎了,好在我领悟到distributedshell就是使用yarn编写的。要研究yarn的话,直接去Hadoop source里面找相应的代码研究即可。
$ hadoop jar hadoop-yarn-applications-distributedshell-2.0.3-alpha.jar --jar hadoop-yarn-applications-distributedshell-2.0.3-alpha.jar org.apache.hadoop.yarn.applications.distributedshell.Client --shell_command uname --shell_args '-a' 13/03/07 21:42:44 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. 13/03/07 21:42:44 INFO distributedshell.Client: Initializing Client 13/03/07 21:42:44 INFO distributedshell.Client: Running Client 13/03/07 21:42:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 13/03/07 21:42:44 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. 13/03/07 21:42:44 INFO distributedshell.Client: Got Cluster metric info from ASM, numNodeManagers=2 13/03/07 21:42:44 INFO distributedshell.Client: Got Cluster node info from ASM 13/03/07 21:42:44 INFO distributedshell.Client: Got node report from ASM for, nodeId=search042091.sqa.cm4:39557, nodeAddresssearch042091.sqa.cm4:8042, nodeRackName/default-rack, nodeNumContainers0, nodeHealthStatusis_node_healthy: true, health_report: "", last_health_report_time: 1362663711950, 13/03/07 21:42:44 INFO distributedshell.Client: Got node report from ASM for, nodeId=search041134.sqa.cm4:49313, nodeAddresssearch041134.sqa.cm4:8042, nodeRackName/default-rack, nodeNumContainers0, nodeHealthStatusis_node_healthy: true, health_report: "", last_health_report_time: 1362663712038, 13/03/07 21:42:44 INFO distributedshell.Client: Queue info, queueName=default, queueCurrentCapacity=0.0, queueMaxCapacity=1.0, queueApplicationCount=17, queueChildQueueCount=0 13/03/07 21:42:44 INFO distributedshell.Client: User ACL Info for Queue, queueName=root, userAcl=SUBMIT_APPLICATIONS 13/03/07 21:42:44 INFO distributedshell.Client: User ACL Info for Queue, queueName=root, userAcl=ADMINISTER_QUEUE 13/03/07 21:42:44 INFO distributedshell.Client: User ACL Info for Queue, queueName=default, userAcl=SUBMIT_APPLICATIONS 13/03/07 21:42:44 INFO distributedshell.Client: User ACL Info for Queue, queueName=default, userAcl=ADMINISTER_QUEUE 13/03/07 21:42:44 INFO distributedshell.Client: Min mem capabililty of resources in this cluster 1024 13/03/07 21:42:44 INFO distributedshell.Client: Max mem capabililty of resources in this cluster 8192 13/03/07 21:42:44 INFO distributedshell.Client: AM memory specified below min threshold of cluster. Using min value., specified=10, min=1024 13/03/07 21:42:44 INFO distributedshell.Client: Setting up application submission context for ASM 13/03/07 21:42:44 INFO distributedshell.Client: Copy App Master jar from local filesystem and add to local environment 13/03/07 21:42:45 INFO distributedshell.Client: Set the environment for the application master 13/03/07 21:42:45 INFO distributedshell.Client: Setting up app master command 13/03/07 21:42:45 INFO distributedshell.Client: Completed setting up app master command ${JAVA_HOME}/bin/java -Xmx1024m org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory 10 --num_containers 1 --priority 0 --shell_command uname --shell_args -a --debug 1><log_dir>/AppMaster.stdout 2><log_dir>/AppMaster.stderr 13/03/07 21:42:45 INFO distributedshell.Client: Submitting application to ASM 13/03/07 21:42:45 INFO client.YarnClientImpl: Submitted application application_1362658309553_0020 to ResourceManager at /10.232.42.91:19001 13/03/07 21:42:46 INFO distributedshell.Client: Got application report from ASM for, appId=20, clientToken=null, appDiagnostics=, appMasterHost=N/A, appQueue=default, appMasterRpcPort=0, appStartTime=1362663765373, yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED, appTrackingUrl=search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0020/, appUser=henshao 13/03/07 21:42:47 INFO distributedshell.Client: Got application report from ASM for, appId=20, clientToken=null, appDiagnostics=, appMasterHost=N/A, appQueue=default, appMasterRpcPort=0, appStartTime=1362663765373, yarnAppState=ACCEPTED, distributedFinalState=UNDEFINED, appTrackingUrl=search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0020/, appUser=henshao 13/03/07 21:42:48 INFO distributedshell.Client: Got application report from ASM for, appId=20, clientToken=null, appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0, appStartTime=1362663765373, yarnAppState=RUNNING, distributedFinalState=UNDEFINED, appTrackingUrl=search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0020/, appUser=henshao 13/03/07 21:42:49 INFO distributedshell.Client: Got application report from ASM for, appId=20, clientToken=null, appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0, appStartTime=1362663765373, yarnAppState=RUNNING, distributedFinalState=UNDEFINED, appTrackingUrl=search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0020/, appUser=henshao 13/03/07 21:42:50 INFO distributedshell.Client: Got application report from ASM for, appId=20, clientToken=null, appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0, appStartTime=1362663765373, yarnAppState=RUNNING, distributedFinalState=UNDEFINED, appTrackingUrl=search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0020/, appUser=henshao 13/03/07 21:42:51 INFO distributedshell.Client: Got application report from ASM for, appId=20, clientToken=null, appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0, appStartTime=1362663765373, yarnAppState=RUNNING, distributedFinalState=UNDEFINED, appTrackingUrl=search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0020/, appUser=henshao 13/03/07 21:42:52 INFO distributedshell.Client: Got application report from ASM for, appId=20, clientToken=null, appDiagnostics=, appMasterHost=, appQueue=default, appMasterRpcPort=0, appStartTime=1362663765373, yarnAppState=FINISHED, distributedFinalState=SUCCEEDED, appTrackingUrl=search042091.sqa.cm4.tbsite.net:8088/proxy/application_1362658309553_0020/, appUser=henshao 13/03/07 21:42:52 INFO distributedshell.Client: Application has completed successfully. Breaking monitoring loop 13/03/07 21:42:52 INFO distributedshell.Client: Application completed successfully </log_dir></log_dir>
运行完成之后,找不到输出在哪儿,费了好大的劲,终于在hadoop/logs/userlogs下面找到输出了。不知道为何运行了两个container。
$ tree hadoop/logs/userlogs/application_1362658309553_0018 application_1362658309553_0018 |-- container_1362658309553_0018_01_000001 | |-- AppMaster.stderr | `-- AppMaster.stdout `-- container_1362658309553_0018_01_000002 |-- stderr `-- stdout $ cat hadoop/logs/userlogs/application_1362658309553_0018/container_1362658309553_0018_01_000002/stdout Linux search042091.sqa.cm4 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
好,开始用yarn调度一个程序。我写了一个脚本,里面启动了服务器。
$ cat ~/start_sp.sh #!/bin/env bash source /home/admin/.bashrc /home/admin/sp/bin/sap_server -c /home/admin/sp/sp_worker/etc/sap_server_app.cfg -l /home/admin/sp/sp_worker/etc/sap_server_log.cfg -k restart
启动起来之后,进程关系图如下。
接着我把脚本直接kill掉,期待yarn给我重启脚本。发现application运行结束了,AppMaster.stderr日志里面有如下内容。
13/03/08 21:40:02 INFO distributedshell.ApplicationMaster: Got response from RM for container ask, completedCnt=1 13/03/08 21:40:02 INFO distributedshell.ApplicationMaster: Got container status for containerID=container_1362747551045_0017_01_000002, state=COMPLETE, exitStatus=137, diagnostics= Killed by external signal 13/03/08 21:40:02 INFO distributedshell.ApplicationMaster: Current application state: loop=464, appDone=true, total=1, requested=1, completed=1, failed=1, currentAllocated=1 13/03/08 21:40:02 INFO distributedshell.ApplicationMaster: Application completed. Signalling finish to RM 13/03/08 21:40:02 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.AMRMClientImpl is stopped. 13/03/08 21:40:02 INFO distributedshell.ApplicationMaster: Application Master failed. exiting
原文地址:Hadoop 2.0配置, 感谢原作者分享。

MySQL 사용자를 추가하는 방법을 마스터하는 것은 데이터베이스 관리자 및 개발자가 데이터베이스의 보안 및 액세스 제어를 보장하기 때문에 데이터베이스 관리자 및 개발자에게 중요합니다. 1) CreateUser 명령을 사용하여 새 사용자를 만듭니다. 2) 보조금 명령을 통해 권한 할당, 3) FlushPrivileges를 사용하여 권한이 적용되도록하십시오.

ChooseCharfixed-lengthdata, varcharforvariable-lengthdata, andtextforlargetextfields.1) charisefficientsconsentent-lengthdatalikecodes.2) varcharsuitsvariable-lengthdatalikeNames, 밸런싱 플렉스 및 성능

MySQL에서 문자열 데이터 유형 및 인덱스를 처리하기위한 모범 사례는 다음과 같습니다. 1) 고정 길이의 Char, 가변 길이의 Varchar 및 큰 텍스트의 텍스트와 같은 적절한 문자열 유형 선택; 2) 인덱싱에 신중하고, 과도한 인덱싱을 피하고, 공통 쿼리에 대한 인덱스를 만듭니다. 3) 접두사 인덱스 및 전체 텍스트 인덱스를 사용하여 긴 문자열 검색을 최적화합니다. 4) 인덱스를 작고 효율적으로 유지하기 위해 인덱스를 정기적으로 모니터링하고 최적화합니다. 이러한 방법을 통해 읽기 및 쓰기 성능의 균형을 맞추고 데이터베이스 효율성을 향상시킬 수 있습니다.

Toaddauserremotelytomysql, 다음에 따르면 : 1) 1) ConnectTomysqlasRoot, 2) CreateEnewerwitHremoteAccess, 3) GrantNecessaryPrivileges 및 4) FlushPrivileges

tostorestringsefficiallyInmysql, choOseTherightDatAtypeBasedOnyOURNEDS : 1) USECHARFIXED-lengthstringsLikeCountryCodes.2) UseVarCharForVariable-lengthstringsLikenames.3) USETEXTFORLONG-FORMTEXTCONTENT.4) USETEXTFORLONG-FORMTEXTCONTENT.4) USETLOBFORBINARYIMAGES

MySQL의 블로브 및 텍스트 데이터 유형을 선택할 때 Blob은 이진 데이터를 저장하는 데 적합하며 텍스트는 텍스트 데이터를 저장하는 데 적합합니다. 1) Blob은 그림 및 오디오와 같은 이진 데이터에 적합합니다. 2) 텍스트는 기사 및 주석과 같은 텍스트 데이터에 적합합니다. 선택할 때는 데이터 속성 및 성능 최적화를 고려해야합니다.

아니요, youshouthusTherootUserInmysqlforyOUrProduct.instead, createScificuserswithlimitedPrivilegestoEnhancesecurity 및 forcuments : 1) grantOnlySerypermissionStothisUser, 3) 정기적으로 재구성 한 사람들이 관리자입니다

mysqlstringdatatatypess는 Bechosenbeasedondatacharacteristicsandusecases : 1) Usecharfixed-lengthstringslikecountryCodes.2) UseVarCharforVariable-lengthstringslikenames.3) UseBaryBarBarBaryBinaryDatalikeCryPyps.4) Usebortextforlargeuns


핫 AI 도구

Undresser.AI Undress
사실적인 누드 사진을 만들기 위한 AI 기반 앱

AI Clothes Remover
사진에서 옷을 제거하는 온라인 AI 도구입니다.

Undress AI Tool
무료로 이미지를 벗다

Clothoff.io
AI 옷 제거제

Video Face Swap
완전히 무료인 AI 얼굴 교환 도구를 사용하여 모든 비디오의 얼굴을 쉽게 바꾸세요!

인기 기사

뜨거운 도구

SublimeText3 영어 버전
권장 사항: Win 버전, 코드 프롬프트 지원!

에디트플러스 중국어 크랙 버전
작은 크기, 구문 강조, 코드 프롬프트 기능을 지원하지 않음

ZendStudio 13.5.1 맥
강력한 PHP 통합 개발 환경

안전한 시험 브라우저
안전한 시험 브라우저는 온라인 시험을 안전하게 치르기 위한 보안 브라우저 환경입니다. 이 소프트웨어는 모든 컴퓨터를 안전한 워크스테이션으로 바꿔줍니다. 이는 모든 유틸리티에 대한 액세스를 제어하고 학생들이 승인되지 않은 리소스를 사용하는 것을 방지합니다.

VSCode Windows 64비트 다운로드
Microsoft에서 출시한 강력한 무료 IDE 편집기