利用下一代Hadoop YARN构建轻量级弹性计算平台-Mysql Tutorial-php.cn

Home

Database

Mysql Tutorial

利用下一代Hadoop YARN构建轻量级弹性计算平台

WBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWB

Jun 07, 2016 pm 04:30 PM

hadoopyarnuseelasticityConstructlightweight

作者:Dong | 新浪微博：西成懂 | 可以转载, 但必须以超链接形式标明文章原始出处和作者信息及版权声明
网址:http://dongxicheng.org/mapreduce-nextgen/use-hadoop-yarn-to-build-a-cloud-system/
本博客的文章集合:http://dongxicheng.org/recommend/

重大消息：我的Hadoop新书《Hadoop技术内幕：深入解析MapReduce架构设计与实现原理》已经开始在各大网站销售了，购书链接地址：当当购书网址，京东购书网址，卓越购书网址。新书官方宣传主页： http://hadoop123.com/。

YARN（Yet Another Resource Negotiator）是下一代Hadoop的一个分支（注意：目前YARN尚不成熟稳定，各大公司普遍使用的还是Hadoop 1.0，但YARN是未来发展趋势，可以提前了解和学习它），它是一个资源管理系统，其上可以运行各种计算框架和应用程序，关于YARN，我已经在多篇文章中进行了介绍:

（1）”相比于MRv1，YARN带来的优势是什么？”

（2）”Hadoop 2.0中的基本术语解释”

（3）”浅谈Borg/YARN/Mesos/Torca/Corona一类系统”

（4）”多集群下资源共享方案介绍”

（5）”解析Google集群资源管理系统Omega”

更多文章可参考：下一代Hadoop YARN全面剖析。

YARN是未来的一个趋势，YARN本身已经变成了一个云操作系统，很多新的计算框架或者应用程序不再基于传统的操作系统开发（比如Linux）,而是基于YARN这个云操作系统，YARN提供了资源管理和资源调度等机制，这意味着，很多新的计算框架或者应用程序脱离了YARN将不再可以单独运行，典型的代表是DAG计算框架Tez和Spark（Spark也可以运行在另一个与YARN类似的资源管理系统Mesos上）。

当前很多计算框架都开发了YARN版本，以能够直接运行在YARN上。本文进行简单的梳理。

（1） MapReduce：MapReduce是一个非常经典的离线计算框架，在MRv1中，MapReduce应用程序运行在由JobTracker和TaskTracker组成的运行时环境中，而在YARN中，不再有JobTracker和TaskTracker这样的服务组件，取而代之的是一个组件ApplicationMaster，它只负责应用程序相关的管理，比如任务切分和调度、任务监控和容错等，而资源相关的调度和管理交给YARN完成。

（2） Tez：Hortonworks开源的DAG计算框架，在MAPREDUCE基础上扩展而来的，重用了MapReduce大量代码，仅支持运行在YARN上，不可单独运行。Tez介绍可参考我的这篇文章：“Apache Tez：一个运行在YARN之上支持DAG作业的计算框架”。

（3）Storm：实时计算框架，运行时环境由Nimbus和supervisor等组件组成，目前storm团队正准备将其移植到YARN上（具体参考：storm plugin for Hadoop YARN：https://groups.google.com/forum/?fromgroups#!topic/storm-user/XBHCau0e5nA），而Yahoo甚至已经开源了一个简单的版本（具体见：https://github.com/yahoo/storm-yarn）。

（4）Spark：Spark是一个基于内存实现的MapReduce计算框架，某些应用场景下更加高效，它目前已经支持运行在YARN上（https://github.com/tweetmagik/spark-yarn）。

随着开源界的发展和推进，最终，YARN之上可以运行各种应用类型的计算框架，包括离线计算框架MapReduce，实时计算框架Storm，DAG计算框架Tez等，真正实现一个集群多用途，这样的集群或者系统，我们通常称为轻量级弹性计算平台，说它轻量级，是因为YARN采用了cgroups轻量级隔离方案，说它弹性，是因为YARN能根据各种计算框架或者应用的负载或者需求调整它们各自占用的资源，实现集群资源共享，资源弹性收缩。在不久的将来，普遍采用的部署方案应该如下：

当然，随着YARN朝着资源管理系统方向更好地发展，最终Impala和Hbase（Hbase已经在做了，参考：https://issues.apache.org/jira/browse/HBASE-4329）这种系统，均可以部署到YARN之上，这样，凡是跟计算沾边的系统均可以部署到YARN上。

原创文章，转载请注明： 转载自董的博客

本文链接地址: http://dongxicheng.org/mapreduce-nextgen/use-hadoop-yarn-to-build-a-cloud-system/

作者：Dong，作者介绍：http://dongxicheng.org/about/

本博客的文章集合:http://dongxicheng.org/recommend/

Copyright © 2013
This feed is for personal, non-commercial use only.
The use of this feed on other websites breaches copyright. If this content is not in your news reader, it makes the page you are viewing an infringement of the copyright. (Digital Fingerprint:
)

Statement

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

How does MySQL index cardinality affect query performance?Apr 14, 2025 am 12:18 AM

MySQL index cardinality has a significant impact on query performance: 1. High cardinality index can more effectively narrow the data range and improve query efficiency; 2. Low cardinality index may lead to full table scanning and reduce query performance; 3. In joint index, high cardinality sequences should be placed in front to optimize query.

MySQL: Resources and Tutorials for New UsersApr 14, 2025 am 12:16 AM

The MySQL learning path includes basic knowledge, core concepts, usage examples, and optimization techniques. 1) Understand basic concepts such as tables, rows, columns, and SQL queries. 2) Learn the definition, working principles and advantages of MySQL. 3) Master basic CRUD operations and advanced usage, such as indexes and stored procedures. 4) Familiar with common error debugging and performance optimization suggestions, such as rational use of indexes and optimization queries. Through these steps, you will have a full grasp of the use and optimization of MySQL.

Real-World MySQL: Examples and Use CasesApr 14, 2025 am 12:15 AM

MySQL's real-world applications include basic database design and complex query optimization. 1) Basic usage: used to store and manage user data, such as inserting, querying, updating and deleting user information. 2) Advanced usage: Handle complex business logic, such as order and inventory management of e-commerce platforms. 3) Performance optimization: Improve performance by rationally using indexes, partition tables and query caches.

SQL Commands in MySQL: Practical ExamplesApr 14, 2025 am 12:09 AM

SQL commands in MySQL can be divided into categories such as DDL, DML, DQL, DCL, etc., and are used to create, modify, delete databases and tables, insert, update, delete data, and perform complex query operations. 1. Basic usage includes CREATETABLE creation table, INSERTINTO insert data, and SELECT query data. 2. Advanced usage involves JOIN for table joins, subqueries and GROUPBY for data aggregation. 3. Common errors such as syntax errors, data type mismatch and permission problems can be debugged through syntax checking, data type conversion and permission management. 4. Performance optimization suggestions include using indexes, avoiding full table scanning, optimizing JOIN operations and using transactions to ensure data consistency.

How does InnoDB handle ACID compliance?Apr 14, 2025 am 12:03 AM

InnoDB achieves atomicity through undolog, consistency and isolation through locking mechanism and MVCC, and persistence through redolog. 1) Atomicity: Use undolog to record the original data to ensure that the transaction can be rolled back. 2) Consistency: Ensure the data consistency through row-level locking and MVCC. 3) Isolation: Supports multiple isolation levels, and REPEATABLEREAD is used by default. 4) Persistence: Use redolog to record modifications to ensure that data is saved for a long time.

MySQL's Place: Databases and ProgrammingApr 13, 2025 am 12:18 AM

MySQL's position in databases and programming is very important. It is an open source relational database management system that is widely used in various application scenarios. 1) MySQL provides efficient data storage, organization and retrieval functions, supporting Web, mobile and enterprise-level systems. 2) It uses a client-server architecture, supports multiple storage engines and index optimization. 3) Basic usages include creating tables and inserting data, and advanced usages involve multi-table JOINs and complex queries. 4) Frequently asked questions such as SQL syntax errors and performance issues can be debugged through the EXPLAIN command and slow query log. 5) Performance optimization methods include rational use of indexes, optimized query and use of caches. Best practices include using transactions and PreparedStatemen

MySQL: From Small Businesses to Large EnterprisesApr 13, 2025 am 12:17 AM

MySQL is suitable for small and large enterprises. 1) Small businesses can use MySQL for basic data management, such as storing customer information. 2) Large enterprises can use MySQL to process massive data and complex business logic to optimize query performance and transaction processing.

What are phantom reads and how does InnoDB prevent them (Next-Key Locking)?Apr 13, 2025 am 12:16 AM

InnoDB effectively prevents phantom reading through Next-KeyLocking mechanism. 1) Next-KeyLocking combines row lock and gap lock to lock records and their gaps to prevent new records from being inserted. 2) In practical applications, by optimizing query and adjusting isolation levels, lock competition can be reduced and concurrency performance can be improved.

See all articles