大数据分析:结合Hadoop或ElasticMapReduce使用Hunk-Mysql Tutorial-php.cn

Home

Database

Mysql Tutorial

大数据分析:结合Hadoop或ElasticMapReduce使用Hunk

WBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWB

Jun 07, 2016 pm 04:10 PM

hadoopdata analysiscombine

作者 Jonathan Allen ，译者张晓鹏 Hunk是Splunk公司一款比较新的产品，用来对Hadoop和其它NoSQL数据存储进行探测和可视化，它的新版本将会支持亚马逊的Elastic MapReduce。结合Hadoop使用Hunk Hadoop由两个单元组成，首先是被称为HDFS的存储单元，HDFS可

作者 Jonathan Allen ，译者张晓鹏

Hunk是Splunk公司一款比较新的产品，用来对Hadoop和其它NoSQL数据存储进行探测和可视化，它的新版本将会支持亚马逊的Elastic MapReduce。

结合Hadoop使用Hunk

Hadoop由两个单元组成，首先是被称为HDFS的存储单元，HDFS可以分布在成千上万个复制的节点上。接下来是MapReduce单元，它负责跟踪和管理被命名为map-reduce jobs的作业。

之前，开发者会用到Splunk Hadoop Connect (SHC)连接器。SHC通过常用的推模型（push model）来输出数据到Hadoop中，这块处理相当地好，但相反方向的处理却可能会有问题。当通过Splunk来探测数据时，原始的数据会被吸收到Splunk Server来检索和处理。就像人们猜想的那样，这个过程并没有发挥出Hadoop计算能力的优势。

Hunk通过提供与Hadoop MapReduce节点协同工作的适配器来解决这个问题。Splunk的查询被转化成Hadoop MapReduce的作业，这些作业在Hadoop集群中处理，最终只有结果被取回到Splunk 服务器中进行分析和可视化。

通过这种方式，Hunk提供了抽象层，以便用户和开发者不需要关心怎么去写Hadoop MapReduce的作业。Hunk还能在MapReduce作业启动前就提供结果预览，以减少无用搜索的数量。

结合Elastic MapReduce使用Hunk

亚马逊的Elastic MapReduce可以看做是对Hadoop的补充，同时也是Hadoop的竞争者。EMR既可以运行在Hadoop HDFS集群上，也可以直接运行在AWS S3上。亚马逊宣称使用AWS S3的优势在于比HDFS集群更易于管理。

当运行Elastic MapReduce时，Hunk提供了相同的抽象层和预览功能，就像它在Hadoop上做的一样。所以从用户的观点来看，在Hadoop和EMR之间切换不会造成什么变化。

云上的Hunk

在云上托管Hunk的传统方法是买一个标准版的许可证，然后部署到虚拟机中，这和你现场安装一样简单。接下来是对Hunk的运行实例进行手动配置以让它对应到正确的Hadoop或AWS集群上。

这个月的新版本里，Hunk的运行实例可以在AWS上进行自动化配置，这包括自动发现EMR数据源，这样Hunk实例可以在几分钟内上线使用。为了充分利用这个优势，Hunk运行实例是按小时来计费。

虚索引（Virtual Indexes）

Hunk中的有个关键概念是“虚索引（Virtual Indexes）”。这些索引已不同原本的意义，变成只是由Hunk来体现Hadoop和EMR集群处理的一种方式。从Splunk的用户界面上看，它们像是真正的索引，即使其数据处理是在map-reduce作业中完成的。并且，由于看起来像索引，你可以在它们之上创建持久的二级索引（persistent secondary indexes）。当你要处理部分数据，然后进一步检查或在多个方面可视化时，这个持久的二级索引会非常有用。

查看英文原文：Big Data Analytics: Using Hunk with Hadoop and Elastic MapReduce

Statement

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Explain the InnoDB Buffer Pool and its importance for performance.Apr 19, 2025 am 12:24 AM

InnoDBBufferPool reduces disk I/O by caching data and indexing pages, improving database performance. Its working principle includes: 1. Data reading: Read data from BufferPool; 2. Data writing: After modifying the data, write to BufferPool and refresh it to disk regularly; 3. Cache management: Use the LRU algorithm to manage cache pages; 4. Reading mechanism: Load adjacent data pages in advance. By sizing the BufferPool and using multiple instances, database performance can be optimized.

MySQL vs. Other Programming Languages: A ComparisonApr 19, 2025 am 12:22 AM

Compared with other programming languages, MySQL is mainly used to store and manage data, while other languages such as Python, Java, and C are used for logical processing and application development. MySQL is known for its high performance, scalability and cross-platform support, suitable for data management needs, while other languages have advantages in their respective fields such as data analytics, enterprise applications, and system programming.

Learning MySQL: A Step-by-Step Guide for New UsersApr 19, 2025 am 12:19 AM

MySQL is worth learning because it is a powerful open source database management system suitable for data storage, management and analysis. 1) MySQL is a relational database that uses SQL to operate data and is suitable for structured data management. 2) The SQL language is the key to interacting with MySQL and supports CRUD operations. 3) The working principle of MySQL includes client/server architecture, storage engine and query optimizer. 4) Basic usage includes creating databases and tables, and advanced usage involves joining tables using JOIN. 5) Common errors include syntax errors and permission issues, and debugging skills include checking syntax and using EXPLAIN commands. 6) Performance optimization involves the use of indexes, optimization of SQL statements and regular maintenance of databases.

MySQL: Essential Skills for Beginners to MasterApr 18, 2025 am 12:24 AM

MySQL is suitable for beginners to learn database skills. 1. Install MySQL server and client tools. 2. Understand basic SQL queries, such as SELECT. 3. Master data operations: create tables, insert, update, and delete data. 4. Learn advanced skills: subquery and window functions. 5. Debugging and optimization: Check syntax, use indexes, avoid SELECT*, and use LIMIT.

MySQL: Structured Data and Relational DatabasesApr 18, 2025 am 12:22 AM

MySQL efficiently manages structured data through table structure and SQL query, and implements inter-table relationships through foreign keys. 1. Define the data format and type when creating a table. 2. Use foreign keys to establish relationships between tables. 3. Improve performance through indexing and query optimization. 4. Regularly backup and monitor databases to ensure data security and performance optimization.

MySQL: Key Features and Capabilities ExplainedApr 18, 2025 am 12:17 AM

MySQL is an open source relational database management system that is widely used in Web development. Its key features include: 1. Supports multiple storage engines, such as InnoDB and MyISAM, suitable for different scenarios; 2. Provides master-slave replication functions to facilitate load balancing and data backup; 3. Improve query efficiency through query optimization and index use.

The Purpose of SQL: Interacting with MySQL DatabasesApr 18, 2025 am 12:12 AM

SQL is used to interact with MySQL database to realize data addition, deletion, modification, inspection and database design. 1) SQL performs data operations through SELECT, INSERT, UPDATE, DELETE statements; 2) Use CREATE, ALTER, DROP statements for database design and management; 3) Complex queries and data analysis are implemented through SQL to improve business decision-making efficiency.

MySQL for Beginners: Getting Started with Database ManagementApr 18, 2025 am 12:10 AM

The basic operations of MySQL include creating databases, tables, and using SQL to perform CRUD operations on data. 1. Create a database: CREATEDATABASEmy_first_db; 2. Create a table: CREATETABLEbooks(idINTAUTO_INCREMENTPRIMARYKEY, titleVARCHAR(100)NOTNULL, authorVARCHAR(100)NOTNULL, published_yearINT); 3. Insert data: INSERTINTObooks(title, author, published_year)VA

See all articles