1.前言 Hadoop-2.4.0 的源码目录下有个 BUILDING.txt 文件，它介绍了如何在 Linux 和 Windows 下编译源代码，本文基本是遵照 BUILDING.txt 指示来操作的，这里再做一下简单的提炼。 第一次编译要求能够访问互联网， Hadoop 的编译依赖非常多的东西，一定要保
1. 前言Hadoop-2.4.0的源码目录下有个BUILDING.txt文件，它介绍了如何在Linux和Windows下编译源代码，本文基本是遵照BUILDING.txt指示来操作的，这里再做一下简单的提炼。
第一次编译要求能够访问互联网，Hadoop的编译依赖非常多的东西，一定要保证机器可访问互联网，否则难逐一解决所有的编译问题，但第一次之后的编译则不用再下载了。
2. 安装依赖
在编译Hadoop 2.4.0源码之前，需要将下列几个依赖的东西安装好：
1) JDK 1.6或更新版本（本文使用JDK1.7，请不要安装JDK1.8版本，JDK1.8和Hadoop 2.4.0不匹配，编译Hadoop 2.4.0源码时会报很多错误）
2) Maven 3.0或更新版本
3) ProtocolBuffer 2.5.0
4) CMake 2.6或更新版本
5) Findbugs 1.3.9，可选的（本文编译时未安装）
在安装好之后，还需要设置一下环境变量，可以修改/etc/profile，也可以是修改~/.profile，增加如下内容：
|
export JAVA_HOME=/root/jdk export CLASSPATH=$JAVA_HOME/lib/tools.jar export PATH=$JAVA_HOME/bin:$PATH
export CMAKE_HOME=/root/cmake export PATH=$CMAKE_HOME/bin:$PATH
export PROTOC_HOME=/root/protobuf export PATH=$PROTOC_HOME/bin:$PATH
export MAVEN_HOME=/root/maven export PATH=$MAVEN_HOME/bin:$PATH |
本文以root用户在/root目录下进行安装，但实际可以选择非root用户及非/root目录进行安装。
2.1. 安装ProtocolBuffer
标准的automake编译安装方式：
1) cd /root
2) tar xzf protobuf-2.5.0.tar.gz
3) cd protobuf-2.5.0
4) ./conigure --prefix=/root/protobuf
5) make
6) make install
2.2. 安装CMake
1) cd /root
2) tar xzf cmake-2.8.12.2.tar.gz
3) cd cmake-2.8.12.2
4) ./bootstrap --prefix=/root/cmake
5) make
6) make install
2.3. 安装JDK
1) cd /root
2) tar xzf jdk-7u55-linux-x64.gz
3) cd jdk1.7.0_55
4) ln -s jdk1.7.0_55 jdk
2.4. 安装Maven
1) cd /root
2) tar xzf apache-maven-3.0.5-bin.tar.gz
3) ln -s apache-maven-3.0.5 maven
3. 编译Hadoop源代码
完成上述准备工作后，即可通过执行命令：mvn package -Pdist -DskipTests -Dtar，启动对Hadoop源代码的编译。请注意一定不要使用JDK1.8。
如果需要编译成本地库（Native Libraries）文件，则使用命令：mvn package -Pdist,native -DskipTests -Dtar。如果C/C++程序需要访问HDFS等，需要使用navite方式编译生成相应的库文件。也可以使用mvn package -Pnative -DskipTests -Dtar特意编译出本地库文件。
相关的编译命令还有：
1) mvn package -Pdist -DskipTests -Dtar
2) mvn package -Pdist,native,docs,src -DskipTests -Dtar
3) mvn package -Psrc -DskipTests
4) mvn package -Pdist,native,docs -DskipTests -Dtar
5) mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site
编译成功后，jar文件会放在target子目录下，可以在Hadoop源码目录下借用find命令搜索各个target子目录。
编译成功后，会生成Hadoop二进制安装包hadoop-2.4.0.tar.gz，放在源代码的hadoop-dist/target子目录下：
main:
[exec] $ tar cf hadoop-2.4.0.tar hadoop-2.4.0
[exec] $ gzip -f hadoop-2.4.0.tar
[exec]
[exec] Hadoop dist tar available at: /root/hadoop-2.4.0-src/hadoop-dist/target/hadoop-2.4.0.tar.gz
[exec]
[INFO] Executed tasks
[INFO]
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /root/hadoop-2.4.0-src/hadoop-dist/target/hadoop-dist-2.4.0-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [4.647s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [5.352s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [7.239s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.424s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [2.918s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [6.261s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [5.321s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [5.953s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [3.783s]
[INFO] Apache Hadoop Common .............................. SUCCESS [1:54.010s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [9.721s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.048s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [4:15.270s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [6:18.553s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [16.237s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [6.543s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.036s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.051s]
[INFO] hadoop-yarn-api ................................... SUCCESS [1:35.227s]
[INFO] hadoop-yarn-common ................................ SUCCESS [43.216s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.055s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [16.476s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [19.942s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [4.926s]
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [9.804s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [23.320s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [1.208s]
[INFO] hadoop-yarn-client ................................ SUCCESS [9.177s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.113s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [4.106s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [3.265s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.056s]
[INFO] hadoop-yarn-project ............................... SUCCESS [5.552s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.096s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [37.231s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [27.135s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [4.886s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [17.876s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [14.140s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [11.305s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [3.083s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [9.855s]
[INFO] hadoop-mapreduce .................................. SUCCESS [5.110s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [7.778s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [12.973s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [3.265s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [11.060s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [7.412s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [4.221s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [4.771s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [0.032s]
[INFO] Apache Hadoop OpenStack support ................... SUCCESS [8.030s]
[INFO] Apache Hadoop Client .............................. SUCCESS [7.730s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.158s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [7.485s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [6.912s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.029s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [40.425s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21:57.892s
[INFO] Finished at: Mon Apr 21 14:33:22 CST 2014
[INFO] Final Memory: 88M/243M
[INFO] ------------------------------------------------------------------------
MySQL's Place: Databases and ProgrammingApr 13, 2025 am 12:18 AMMySQL's position in databases and programming is very important. It is an open source relational database management system that is widely used in various application scenarios. 1) MySQL provides efficient data storage, organization and retrieval functions, supporting Web, mobile and enterprise-level systems. 2) It uses a client-server architecture, supports multiple storage engines and index optimization. 3) Basic usages include creating tables and inserting data, and advanced usages involve multi-table JOINs and complex queries. 4) Frequently asked questions such as SQL syntax errors and performance issues can be debugged through the EXPLAIN command and slow query log. 5) Performance optimization methods include rational use of indexes, optimized query and use of caches. Best practices include using transactions and PreparedStatemen
MySQL: From Small Businesses to Large EnterprisesApr 13, 2025 am 12:17 AMMySQL is suitable for small and large enterprises. 1) Small businesses can use MySQL for basic data management, such as storing customer information. 2) Large enterprises can use MySQL to process massive data and complex business logic to optimize query performance and transaction processing.
What are phantom reads and how does InnoDB prevent them (Next-Key Locking)?Apr 13, 2025 am 12:16 AMInnoDB effectively prevents phantom reading through Next-KeyLocking mechanism. 1) Next-KeyLocking combines row lock and gap lock to lock records and their gaps to prevent new records from being inserted. 2) In practical applications, by optimizing query and adjusting isolation levels, lock competition can be reduced and concurrency performance can be improved.
MySQL: Not a Programming Language, But...Apr 13, 2025 am 12:03 AMMySQL is not a programming language, but its query language SQL has the characteristics of a programming language: 1. SQL supports conditional judgment, loops and variable operations; 2. Through stored procedures, triggers and functions, users can perform complex logical operations in the database.
MySQL: An Introduction to the World's Most Popular DatabaseApr 12, 2025 am 12:18 AMMySQL is an open source relational database management system, mainly used to store and retrieve data quickly and reliably. Its working principle includes client requests, query resolution, execution of queries and return results. Examples of usage include creating tables, inserting and querying data, and advanced features such as JOIN operations. Common errors involve SQL syntax, data types, and permissions, and optimization suggestions include the use of indexes, optimized queries, and partitioning of tables.
The Importance of MySQL: Data Storage and ManagementApr 12, 2025 am 12:18 AMMySQL is an open source relational database management system suitable for data storage, management, query and security. 1. It supports a variety of operating systems and is widely used in Web applications and other fields. 2. Through the client-server architecture and different storage engines, MySQL processes data efficiently. 3. Basic usage includes creating databases and tables, inserting, querying and updating data. 4. Advanced usage involves complex queries and stored procedures. 5. Common errors can be debugged through the EXPLAIN statement. 6. Performance optimization includes the rational use of indexes and optimized query statements.
Why Use MySQL? Benefits and AdvantagesApr 12, 2025 am 12:17 AMMySQL is chosen for its performance, reliability, ease of use, and community support. 1.MySQL provides efficient data storage and retrieval functions, supporting multiple data types and advanced query operations. 2. Adopt client-server architecture and multiple storage engines to support transaction and query optimization. 3. Easy to use, supports a variety of operating systems and programming languages. 4. Have strong community support and provide rich resources and solutions.
Describe InnoDB locking mechanisms (shared locks, exclusive locks, intention locks, record locks, gap locks, next-key locks).Apr 12, 2025 am 12:16 AMInnoDB's lock mechanisms include shared locks, exclusive locks, intention locks, record locks, gap locks and next key locks. 1. Shared lock allows transactions to read data without preventing other transactions from reading. 2. Exclusive lock prevents other transactions from reading and modifying data. 3. Intention lock optimizes lock efficiency. 4. Record lock lock index record. 5. Gap lock locks index recording gap. 6. The next key lock is a combination of record lock and gap lock to ensure data consistency.
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
WebStorm Mac version
Useful JavaScript development tools
Zend Studio 13.0.1
Powerful PHP integrated development environment
DVWA
Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software
Atom editor mac version download
The most popular open source editor
Dreamweaver CS6
Visual web development tools
