search
HomeDatabaseMysql Tutorialhadoop第一个程序WordCount.java的编译运行过程

java是hadoop开发的标准官方语言,本文下载了官方的WordCount.java并对其进行了编译和打包,然后使用测试数据运行了该hadoop程序。 这里假定已经装好了hadoop的环境,在Linux下运行hadoop命令能够正常执行; 下载java版本的WordCount.java程序。 将WordCount

java是hadoop开发的标准官方语言,本文下载了官方的WordCount.java并对其进行了编译和打包,然后使用测试数据运行了该hadoop程序。

这里假定已经装好了hadoop的环境,在Linux下运行hadoop命令能够正常执行;

下载java版本的WordCount.java程序。

将WordCount.java复制到linux下的一个目录,这里我复制到/home/crazyant/hadoop_wordcount

[crazyant@dev.mechine hadoop_wordcount]$ ll

total 4

-rwxr–r–? 1 crazyant crazyant 1921 Aug 16 20:03 WordCount.java

在该目录(/home/crazyant/hadoop_wordcount)下创建wordcount_classes目录,用于存放编译WordCount.java生成的class文件。

[crazyant@dev.mechine hadoop_wordcount]$ mkdir wordcount_classes

[crazyant@dev.mechine hadoop_wordcount]$ ll

total 8

drwxrwxr-x? 2 crazyant crazyant 4096 Aug 16 20:07 wordcount_classes

-rwxr–r–? 1 crazyant crazyant 1921 Aug 16 20:03 WordCount.java

编译WordCount.java文件,其中-classpath选项表示要引用hadoop官方的包,-d选项表示要将编译后的class文件生成的目标目录。

[crazyant@dev.mechine hadoop_wordcount]$ javac -classpath /home/crazyant/app/hadoop/hadoop-2-core.jar -d wordcount_classes WordCount.java

[crazyant@dev.mechine hadoop_wordcount]$ ll -R

.:

total 8

drwxrwxr-x? 3 crazyant crazyant 4096 Aug 16 20:09 wordcount_classes

-rwxr–r–? 1 crazyant crazyant 1921 Aug 16 20:03 WordCount.java

./wordcount_classes:

total 4

drwxrwxr-x? 3 crazyant crazyant 4096 Aug 16 20:09 org

./wordcount_classes/org:

total 4

drwxrwxr-x? 2 crazyant crazyant 4096 Aug 16 20:09 myorg

./wordcount_classes/org/myorg:

total 12

-rw-rw-r–? 1 crazyant crazyant 1546 Aug 16 20:09 WordCount.class

-rw-rw-r–? 1 crazyant crazyant 1938 Aug 16 20:09 WordCount$Map.class

-rw-rw-r–? 1 crazyant crazyant 1611 Aug 16 20:09 WordCount$Reduce.class

然后将编译后的class文件打包:

[crazyant@dev.mechine hadoop_wordcount]$ jar -cvf wordcount.jar -C wordcount_classes/ .

added manifest

adding: org/(in = 0) (out= 0)(stored 0%)

adding: org/myorg/(in = 0) (out= 0)(stored 0%)

adding: org/myorg/WordCount$Map.class(in = 1938) (out= 798)(deflated 58%)

adding: org/myorg/WordCount$Reduce.class(in = 1611) (out= 649)(deflated 59%)

adding: org/myorg/WordCount.class(in = 1546) (out= 749)(deflated 51%)

[crazyant@dev.mechine hadoop_wordcount]$ ll

total 12

drwxrwxr-x? 3 crazyant crazyant 4096 Aug 16 20:09 wordcount_classes

-rw-rw-r–? 1 crazyant crazyant 3169 Aug 16 20:11 wordcount.jar

-rwxr–r–? 1 crazyant crazyant 1921 Aug 16 20:03 WordCount.java

在本地用echo生成一个文件,用于输入数据:

[crazyant@dev.mechine hadoop_wordcount]$ echo “hello world, hello crazyant, i am the ant, i am your brother” > inputfile

[crazyant@dev.mechine hadoop_wordcount]$ more inputfile

hello world, hello crazyant, i am the ant, i am your brother

在hadoop上建立一个目录,里面建立输入文件的目录

[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -mkdir /app/word_count/input

[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -ls /app/word_count

Found 1 items

drwxr-xr-x?? 3 czt czt????????? 0 2013-08-16 20:16 /app/word_count/input

将本地刚刚写的的inputfile上传到hadoop上的input目录

[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -put inputfile /app/word_count/input

[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -ls /app/word_count/input

Found 1 items

-rw-r–r–?? 3 czt czt???????? 61 2013-08-16 20:18 /app/word_count/input/inputfile

运行jar,以建立的Input目录作为输入参数

[crazyant@dev.mechine hadoop_wordcount]$ hadoop jar wordcount.jar org.myorg.WordCount /app/word_count/input /app/word_count/output

13/08/16 20:19:38 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

13/08/16 20:19:40 INFO util.NativeCodeLoader: Loaded the native-hadoop library

13/08/16 20:19:40 INFO compress.LzoCodec: Successfully loaded & initialized native-lzo library

13/08/16 20:19:40 INFO compress.LzmaCodec: Successfully loaded & initialized native-lzma library

13/08/16 20:19:40 INFO compress.QuickLzCodec: Successfully loaded & initialized native-quicklz library

13/08/16 20:19:40 INFO mapred.FileInputFormat: Total input paths to process : 1

13/08/16 20:19:41 INFO mapred.JobClient: splits size : 61

13/08/16 20:19:41 INFO mapred.JobClient: Running job: job_20130813122541_105844

13/08/16 20:19:43 INFO mapred.JobClient:? map 0% reduce 0%

13/08/16 20:19:57 INFO mapred.JobClient:? map 24% reduce 0%

13/08/16 20:20:07 INFO mapred.JobClient:? map 93% reduce 0%

13/08/16 20:20:16 INFO mapred.JobClient:? map 100% reduce 1%

13/08/16 20:20:26 INFO mapred.JobClient:? map 100% reduce 61%

13/08/16 20:20:36 INFO mapred.JobClient:? map 100% reduce 89%

13/08/16 20:20:47 INFO mapred.JobClient:? map 100% reduce 96%

13/08/16 20:20:57 INFO mapred.JobClient:? map 100% reduce 98%

13/08/16 20:21:00 INFO mapred.JobClient: Updating completed job! Ignoring …

13/08/16 20:21:00 INFO mapred.JobClient: Updating completed job! Ignoring …

13/08/16 20:21:00 INFO mapred.JobClient: Job complete: job_20130813122541_105844

13/08/16 20:21:00 INFO mapred.JobClient: Counters: 19

13/08/16 20:21:00 INFO mapred.JobClient:?? File Systems

13/08/16 20:21:00 INFO mapred.JobClient:???? HDFS bytes read=1951

13/08/16 20:21:00 INFO mapred.JobClient:???? HDFS bytes written=68

13/08/16 20:21:00 INFO mapred.JobClient:???? Local bytes read=5174715

13/08/16 20:21:00 INFO mapred.JobClient:???? Local bytes written=256814

13/08/16 20:21:00 INFO mapred.JobClient:?? Job Counters

13/08/16 20:21:00 INFO mapred.JobClient:???? Launched reduce tasks=100

13/08/16 20:21:00 INFO mapred.JobClient:???? Rack-local map tasks=61

13/08/16 20:21:00 INFO mapred.JobClient:???? ORIGINAL_REDUCES=100

13/08/16 20:21:00 INFO mapred.JobClient:???? Launched map tasks=61

13/08/16 20:21:00 INFO mapred.JobClient:???? MISS_SCHEDULED_REDUCES=15

13/08/16 20:21:00 INFO mapred.JobClient:?? TASK_STATISTICS

13/08/16 20:21:00 INFO mapred.JobClient:???? Total Map Slot Time=34

13/08/16 20:21:00 INFO mapred.JobClient:???? Attempt_0 Map Task Count=61

13/08/16 20:21:00 INFO mapred.JobClient:???? Total Reduce Slot Time=892

13/08/16 20:21:00 INFO mapred.JobClient:?? Map-Reduce Framework

13/08/16 20:21:00 INFO mapred.JobClient:???? Reduce input groups=9

13/08/16 20:21:00 INFO mapred.JobClient:???? Combine output records=0

13/08/16 20:21:00 INFO mapred.JobClient:???? Map input records=1

13/08/16 20:21:00 INFO mapred.JobClient:???? Reduce output records=9

13/08/16 20:21:00 INFO mapred.JobClient:???? Map input bytes=61

13/08/16 20:21:00 INFO mapred.JobClient:???? Combine input records=0

13/08/16 20:21:00 INFO mapred.JobClient:???? Reduce input records=9

查看output目录是否有结果

[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -ls /app/word_count/output??????????????????????????????????????????????????? Found 100 items

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00000

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00001

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00002

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00003

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00004

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00005

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00006

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00007

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00008

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00009

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00010

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00011

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00012

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00013

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00014

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00015

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00016

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00017

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00018

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00019

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00020

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00021

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00022

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00023

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00024

-rw-r–r–?? 3 czt czt????????? 0 2013-08-16 20:20 /app/word_count/output/part-00025

将该目录下所有文本文件合并后下载到本地

[crazyant@dev.mechine hadoop_wordcount]$ hadoop fs -getmerge /app/word_count/output wordcount_result

[crazyant@dev.mechine hadoop_wordcount]$ ls

inputfile? wordcount_classes? wordcount.jar? WordCount.java? wordcount_result

查看一下下载下来的计算结果

[crazyant@dev.mechine hadoop_wordcount]$ more wordcount_result

i?????? 2

your??? 1

crazyant,?????? 1

brother 1

hello?? 2

am????? 2

world,? 1

the???? 1

ant,??? 1

统计结果正确;

参考文章:http://hadoop.apache.org/docs/r0.18.3/mapred_tutorial.html#Example%3A+WordCount+v1.0

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Explain the InnoDB Buffer Pool and its importance for performance.Explain the InnoDB Buffer Pool and its importance for performance.Apr 19, 2025 am 12:24 AM

InnoDBBufferPool reduces disk I/O by caching data and indexing pages, improving database performance. Its working principle includes: 1. Data reading: Read data from BufferPool; 2. Data writing: After modifying the data, write to BufferPool and refresh it to disk regularly; 3. Cache management: Use the LRU algorithm to manage cache pages; 4. Reading mechanism: Load adjacent data pages in advance. By sizing the BufferPool and using multiple instances, database performance can be optimized.

MySQL vs. Other Programming Languages: A ComparisonMySQL vs. Other Programming Languages: A ComparisonApr 19, 2025 am 12:22 AM

Compared with other programming languages, MySQL is mainly used to store and manage data, while other languages ​​such as Python, Java, and C are used for logical processing and application development. MySQL is known for its high performance, scalability and cross-platform support, suitable for data management needs, while other languages ​​have advantages in their respective fields such as data analytics, enterprise applications, and system programming.

Learning MySQL: A Step-by-Step Guide for New UsersLearning MySQL: A Step-by-Step Guide for New UsersApr 19, 2025 am 12:19 AM

MySQL is worth learning because it is a powerful open source database management system suitable for data storage, management and analysis. 1) MySQL is a relational database that uses SQL to operate data and is suitable for structured data management. 2) The SQL language is the key to interacting with MySQL and supports CRUD operations. 3) The working principle of MySQL includes client/server architecture, storage engine and query optimizer. 4) Basic usage includes creating databases and tables, and advanced usage involves joining tables using JOIN. 5) Common errors include syntax errors and permission issues, and debugging skills include checking syntax and using EXPLAIN commands. 6) Performance optimization involves the use of indexes, optimization of SQL statements and regular maintenance of databases.

MySQL: Essential Skills for Beginners to MasterMySQL: Essential Skills for Beginners to MasterApr 18, 2025 am 12:24 AM

MySQL is suitable for beginners to learn database skills. 1. Install MySQL server and client tools. 2. Understand basic SQL queries, such as SELECT. 3. Master data operations: create tables, insert, update, and delete data. 4. Learn advanced skills: subquery and window functions. 5. Debugging and optimization: Check syntax, use indexes, avoid SELECT*, and use LIMIT.

MySQL: Structured Data and Relational DatabasesMySQL: Structured Data and Relational DatabasesApr 18, 2025 am 12:22 AM

MySQL efficiently manages structured data through table structure and SQL query, and implements inter-table relationships through foreign keys. 1. Define the data format and type when creating a table. 2. Use foreign keys to establish relationships between tables. 3. Improve performance through indexing and query optimization. 4. Regularly backup and monitor databases to ensure data security and performance optimization.

MySQL: Key Features and Capabilities ExplainedMySQL: Key Features and Capabilities ExplainedApr 18, 2025 am 12:17 AM

MySQL is an open source relational database management system that is widely used in Web development. Its key features include: 1. Supports multiple storage engines, such as InnoDB and MyISAM, suitable for different scenarios; 2. Provides master-slave replication functions to facilitate load balancing and data backup; 3. Improve query efficiency through query optimization and index use.

The Purpose of SQL: Interacting with MySQL DatabasesThe Purpose of SQL: Interacting with MySQL DatabasesApr 18, 2025 am 12:12 AM

SQL is used to interact with MySQL database to realize data addition, deletion, modification, inspection and database design. 1) SQL performs data operations through SELECT, INSERT, UPDATE, DELETE statements; 2) Use CREATE, ALTER, DROP statements for database design and management; 3) Complex queries and data analysis are implemented through SQL to improve business decision-making efficiency.

MySQL for Beginners: Getting Started with Database ManagementMySQL for Beginners: Getting Started with Database ManagementApr 18, 2025 am 12:10 AM

The basic operations of MySQL include creating databases, tables, and using SQL to perform CRUD operations on data. 1. Create a database: CREATEDATABASEmy_first_db; 2. Create a table: CREATETABLEbooks(idINTAUTO_INCREMENTPRIMARYKEY, titleVARCHAR(100)NOTNULL, authorVARCHAR(100)NOTNULL, published_yearINT); 3. Insert data: INSERTINTObooks(title, author, published_year)VA

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

DVWA

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

Safe Exam Browser

Safe Exam Browser

Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

SublimeText3 Linux new version

SublimeText3 Linux new version

SublimeText3 Linux latest version

mPDF

mPDF

mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),