MySQL向Hive/HBase的迁移工具-Mysql Tutorial-php.cn

Home

Database

Mysql Tutorial

MySQL向Hive/HBase的迁移工具

WBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWB

Jun 07, 2016 pm 04:26 PM

apachehbasehivemysqltoolmigrate

Apache Hive是目前大型数据仓库的免费首选产品之一，使用Apache Hive的人是不会期望在小数据量上做什么文章，例如把MySQL中的数据搬到Hive/HBase中去，那样的话原先很快能执行完毕的SQL，估计在Hive上运行跟原来相比时间延长10倍都不止。但如果你有MySQL数据

Apache Hive是目前大型数据仓库的免费首选产品之一，使用Apache Hive的人是不会期望在小数据量上做什么文章，例如把MySQL中的数据搬到Hive/HBase中去，那样的话原先很快能执行完毕的SQL，估计在Hive上运行跟原来相比时间延长10倍都不止。但如果你有MySQL数据可以把大量的数据向Hive导入，如果上亿条的数据量再加上复杂的SQL查询条件对于MySQL来说是一件比较头疼的事情，此时相比而言对于Hive来说还算比较easy没有那么非常的头痛，但是两者之间缺少一个沟通的桥梁。

而然伟大的云计算公司cloudera.com也是Hadoop强力支持者推出了Sqoop，Sqoop顾名思义SQL-to-Hadoop，在sqoop中通过 ManagerFactory 抽象类对多种数据库类型进行了抽象，可以做到 Hsqldb、MySQL、Oracle、PostgreSQL 这些数据库中的数据可以向Hive中写入。

从导出/导入所有数据一条命令即可，而且可以对表和数据的筛选，开发的效率提升和配置的简洁是这个工具的特色所在，同样的机器配置、机器数量、数据量和数据内容，但是换了不同的环境得到了不同的执行效率，通过对RMDBS到Hadoop的迁移，带来了性能的提升，所以就体现了sqoop的价值。

在一次开发大会上提到的Sqoop主要功能
    JDBC-based implementation
        ? Works with many popular database vendors
    Auto-generation of tedious user-side code
        ? Write MapReduce applications to work with your data, faster
    Integration with Hive
        ? Allows you to stay in a SQL-based environment
    Extensible backend
        ? Database-specific code paths for better performance

具体操作手册相见：
http://archive.cloudera.com/cdh/3/sqoop/SqoopUserGuide.html (官方)

相关文章：
Hive入门3–Hive与HBase的整合
Apache Hive入门2
Apache Hive入门1
Apache Pig入门1 –介绍/基本架构/与Hive对比

–end–

原文地址：MySQL向Hive/HBase的迁移工具, 感谢原作者分享。

Statement

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

What are stored procedures in MySQL?May 01, 2025 am 12:27 AM

Stored procedures are precompiled SQL statements in MySQL for improving performance and simplifying complex operations. 1. Improve performance: After the first compilation, subsequent calls do not need to be recompiled. 2. Improve security: Restrict data table access through permission control. 3. Simplify complex operations: combine multiple SQL statements to simplify application layer logic.

How does query caching work in MySQL?May 01, 2025 am 12:26 AM

The working principle of MySQL query cache is to store the results of SELECT query, and when the same query is executed again, the cached results are directly returned. 1) Query cache improves database reading performance and finds cached results through hash values. 2) Simple configuration, set query_cache_type and query_cache_size in MySQL configuration file. 3) Use the SQL_NO_CACHE keyword to disable the cache of specific queries. 4) In high-frequency update environments, query cache may cause performance bottlenecks and needs to be optimized for use through monitoring and adjustment of parameters.

What are the advantages of using MySQL over other relational databases?May 01, 2025 am 12:18 AM

The reasons why MySQL is widely used in various projects include: 1. High performance and scalability, supporting multiple storage engines; 2. Easy to use and maintain, simple configuration and rich tools; 3. Rich ecosystem, attracting a large number of community and third-party tool support; 4. Cross-platform support, suitable for multiple operating systems.

How do you handle database upgrades in MySQL?Apr 30, 2025 am 12:28 AM

The steps for upgrading MySQL database include: 1. Backup the database, 2. Stop the current MySQL service, 3. Install the new version of MySQL, 4. Start the new version of MySQL service, 5. Recover the database. Compatibility issues are required during the upgrade process, and advanced tools such as PerconaToolkit can be used for testing and optimization.

What are the different backup strategies you can use for MySQL?Apr 30, 2025 am 12:28 AM

MySQL backup policies include logical backup, physical backup, incremental backup, replication-based backup, and cloud backup. 1. Logical backup uses mysqldump to export database structure and data, which is suitable for small databases and version migrations. 2. Physical backups are fast and comprehensive by copying data files, but require database consistency. 3. Incremental backup uses binary logging to record changes, which is suitable for large databases. 4. Replication-based backup reduces the impact on the production system by backing up from the server. 5. Cloud backups such as AmazonRDS provide automation solutions, but costs and control need to be considered. When selecting a policy, database size, downtime tolerance, recovery time, and recovery point goals should be considered.

What is MySQL clustering?Apr 30, 2025 am 12:28 AM

MySQLclusteringenhancesdatabaserobustnessandscalabilitybydistributingdataacrossmultiplenodes.ItusestheNDBenginefordatareplicationandfaulttolerance,ensuringhighavailability.Setupinvolvesconfiguringmanagement,data,andSQLnodes,withcarefulmonitoringandpe

How do you optimize database schema design for performance in MySQL?Apr 30, 2025 am 12:27 AM

Optimizing database schema design in MySQL can improve performance through the following steps: 1. Index optimization: Create indexes on common query columns, balancing the overhead of query and inserting updates. 2. Table structure optimization: Reduce data redundancy through normalization or anti-normalization and improve access efficiency. 3. Data type selection: Use appropriate data types, such as INT instead of VARCHAR, to reduce storage space. 4. Partitioning and sub-table: For large data volumes, use partitioning and sub-table to disperse data to improve query and maintenance efficiency.

How can you optimize MySQL performance?Apr 30, 2025 am 12:26 AM

TooptimizeMySQLperformance,followthesesteps:1)Implementproperindexingtospeedupqueries,2)UseEXPLAINtoanalyzeandoptimizequeryperformance,3)Adjustserverconfigurationsettingslikeinnodb_buffer_pool_sizeandmax_connections,4)Usepartitioningforlargetablestoi

See all articles