search
HomeJavajavaTutorialSpecific analysis of Java custom OutputFormat

Specific analysis of Java custom OutputFormat

Aug 20, 2017 am 09:33 AM
javacustomize

This article mainly introduces relevant information about the detailed explanation of custom OutputFormat examples in java. Here are examples to help you learn and understand this part of the content. I hope this article can help everyone. Friends in need can refer to it

Detailed explanation of examples of custom OutputFormat in java

Example code:


package com.ccse.hadoop.outputformat; 
 
import java.io.IOException; 
import java.net.URI; 
import java.net.URISyntaxException; 
import java.util.StringTokenizer; 
 
import org.apache.hadoop.conf.Configuration; 
import org.apache.hadoop.fs.FSDataOutputStream; 
import org.apache.hadoop.fs.FileSystem; 
import org.apache.hadoop.fs.Path; 
import org.apache.hadoop.io.LongWritable; 
import org.apache.hadoop.io.Text; 
import org.apache.hadoop.mapreduce.Job; 
import org.apache.hadoop.mapreduce.JobContext; 
import org.apache.hadoop.mapreduce.Mapper; 
import org.apache.hadoop.mapreduce.OutputCommitter; 
import org.apache.hadoop.mapreduce.OutputFormat; 
import org.apache.hadoop.mapreduce.RecordWriter; 
import org.apache.hadoop.mapreduce.Reducer; 
import org.apache.hadoop.mapreduce.TaskAttemptContext; 
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 
import org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter; 
 
 
public class MySelfOutputFormatApp { 
   
  public final static String INPUT_PATH = "hdfs://chaoren1:9000/mapinput"; 
  public final static String OUTPUT_PATH = "hdfs://chaoren1:9000/mapoutput"; 
  public final static String OUTPUT_FILENAME = "/abc"; 
   
  public static void main(String[] args) throws IOException, URISyntaxException,  
    ClassNotFoundException, InterruptedException { 
    Configuration conf = new Configuration(); 
    FileSystem fileSystem = FileSystem.get(new URI(OUTPUT_PATH), conf); 
    fileSystem.delete(new Path(OUTPUT_PATH), true); 
     
    Job job = new Job(conf, MySelfOutputFormatApp.class.getSimpleName()); 
    job.setJarByClass(MySelfOutputFormatApp.class); 
     
    FileInputFormat.setInputPaths(job, new Path(INPUT_PATH)); 
    job.setMapperClass(MyMapper.class); 
    job.setMapOutputKeyClass(Text.class); 
    job.setMapOutputValueClass(LongWritable.class); 
     
    job.setReducerClass(MyReducer.class); 
    job.setOutputKeyClass(Text.class); 
    job.setOutputValueClass(LongWritable.class); 
    job.setOutputFormatClass(MyselfOutputFormat.class); 
     
    job.waitForCompletion(true); 
  } 
   
  public static class MyMapper extends Mapper<LongWritable, Text, Text, LongWritable> { 
 
    private Text word = new Text(); 
    private LongWritable writable = new LongWritable(1); 
     
    @Override 
    protected void map(LongWritable key, Text value, 
        Mapper<LongWritable, Text, Text, LongWritable>.Context context) 
        throws IOException, InterruptedException { 
      if (value != null) { 
        String line = value.toString(); 
        StringTokenizer tokenizer = new StringTokenizer(line); 
        while (tokenizer.hasMoreTokens()) { 
          word.set(tokenizer.nextToken()); 
          context.write(word, writable); 
        } 
      } 
    } 
     
  } 
   
  public static class MyReducer extends Reducer<Text, LongWritable, Text, LongWritable> { 
 
    @Override 
    protected void reduce(Text key, Iterable<LongWritable> values, 
        Reducer<Text, LongWritable, Text, LongWritable>.Context context) 
        throws IOException, InterruptedException { 
      long sum = 0;  
      for (LongWritable value : values) { 
        sum += value.get(); 
      } 
      context.write(key, new LongWritable(sum)); 
    } 
  } 
 
  public static class MyselfOutputFormat extends OutputFormat<Text, LongWritable> { 
 
    private FSDataOutputStream outputStream = null; 
     
    @Override 
    public RecordWriter<Text, LongWritable> getRecordWriter( 
        TaskAttemptContext context) throws IOException, 
        InterruptedException { 
      try { 
        FileSystem fileSystem = FileSystem.get(new URI(MySelfOutputFormatApp.OUTPUT_PATH), context.getConfiguration()); 
        //指定文件的输出路径 
        final Path path = new Path(MySelfOutputFormatApp.OUTPUT_PATH  
                     + MySelfOutputFormatApp.OUTPUT_FILENAME); 
        this.outputStream = fileSystem.create(path, false); 
      } catch (URISyntaxException e) { 
        e.printStackTrace(); 
      } 
      return new MySelfRecordWriter(outputStream); 
    } 
 
    @Override 
    public void checkOutputSpecs(JobContext context) throws IOException, 
        InterruptedException { 
    } 
 
    @Override 
    public OutputCommitter getOutputCommitter(TaskAttemptContext context) 
        throws IOException, InterruptedException { 
      return new FileOutputCommitter(new Path(MySelfOutputFormatApp.OUTPUT_PATH), context); 
    } 
     
  } 
   
  public static class MySelfRecordWriter extends RecordWriter<Text, LongWritable> { 
 
    private FSDataOutputStream outputStream = null; 
     
    public MySelfRecordWriter(FSDataOutputStream outputStream) { 
      this.outputStream = outputStream; 
    } 
     
    @Override 
    public void write(Text key, LongWritable value) throws IOException, 
        InterruptedException { 
      this.outputStream.writeBytes(key.toString()); 
      this.outputStream.writeBytes("\t"); 
      this.outputStream.writeLong(value.get()); 
    } 
 
    @Override 
    public void close(TaskAttemptContext context) throws IOException, 
        InterruptedException { 
      this.outputStream.close(); 
    } 
     
  } 
   
}

2.OutputFormat is used Handles various output destinations.

2.1 The key-value pairs that OutputFormat needs to write out come from the Reducer class and are obtained through RecordWriter.

2.2 The write(...) method in RecordWriter only has k and v. Where should it be written? This is handled by passing in the OutputStream separately. write is to write k and v into the OutputStream.

2.3 RecordWriter class is located in OutputFormat. Therefore, our custom OutputFromat must inherit the OutputFormat type. Then, the stream object must be obtained in the getRecordWriter(...) method.

The above is the detailed content of Specific analysis of Java custom OutputFormat. For more information, please follow other related articles on the PHP Chinese website!

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
What aspects of Java development are platform-dependent?What aspects of Java development are platform-dependent?Apr 26, 2025 am 12:19 AM

Javadevelopmentisnotentirelyplatform-independentduetoseveralfactors.1)JVMvariationsaffectperformanceandbehavioracrossdifferentOS.2)NativelibrariesviaJNIintroduceplatform-specificissues.3)Filepathsandsystempropertiesdifferbetweenplatforms.4)GUIapplica

Are there performance differences when running Java code on different platforms? Why?Are there performance differences when running Java code on different platforms? Why?Apr 26, 2025 am 12:15 AM

Java code will have performance differences when running on different platforms. 1) The implementation and optimization strategies of JVM are different, such as OracleJDK and OpenJDK. 2) The characteristics of the operating system, such as memory management and thread scheduling, will also affect performance. 3) Performance can be improved by selecting the appropriate JVM, adjusting JVM parameters and code optimization.

What are some limitations of Java's platform independence?What are some limitations of Java's platform independence?Apr 26, 2025 am 12:10 AM

Java'splatformindependencehaslimitationsincludingperformanceoverhead,versioncompatibilityissues,challengeswithnativelibraryintegration,platform-specificfeatures,andJVMinstallation/maintenance.Thesefactorscomplicatethe"writeonce,runanywhere"

Explain the difference between platform independence and cross-platform development.Explain the difference between platform independence and cross-platform development.Apr 26, 2025 am 12:08 AM

Platformindependenceallowsprogramstorunonanyplatformwithoutmodification,whilecross-platformdevelopmentrequiressomeplatform-specificadjustments.Platformindependence,exemplifiedbyJava,enablesuniversalexecutionbutmaycompromiseperformance.Cross-platformd

How does Just-In-Time (JIT) compilation affect Java's performance and platform independence?How does Just-In-Time (JIT) compilation affect Java's performance and platform independence?Apr 26, 2025 am 12:02 AM

JITcompilationinJavaenhancesperformancewhilemaintainingplatformindependence.1)Itdynamicallytranslatesbytecodeintonativemachinecodeatruntime,optimizingfrequentlyusedcode.2)TheJVMremainsplatform-independent,allowingthesameJavaapplicationtorunondifferen

Why is Java a popular choice for developing cross-platform desktop applications?Why is Java a popular choice for developing cross-platform desktop applications?Apr 25, 2025 am 12:23 AM

Javaispopularforcross-platformdesktopapplicationsduetoits"WriteOnce,RunAnywhere"philosophy.1)ItusesbytecodethatrunsonanyJVM-equippedplatform.2)LibrarieslikeSwingandJavaFXhelpcreatenative-lookingUIs.3)Itsextensivestandardlibrarysupportscompr

Discuss situations where writing platform-specific code in Java might be necessary.Discuss situations where writing platform-specific code in Java might be necessary.Apr 25, 2025 am 12:22 AM

Reasons for writing platform-specific code in Java include access to specific operating system features, interacting with specific hardware, and optimizing performance. 1) Use JNA or JNI to access the Windows registry; 2) Interact with Linux-specific hardware drivers through JNI; 3) Use Metal to optimize gaming performance on macOS through JNI. Nevertheless, writing platform-specific code can affect the portability of the code, increase complexity, and potentially pose performance overhead and security risks.

What are the future trends in Java development that relate to platform independence?What are the future trends in Java development that relate to platform independence?Apr 25, 2025 am 12:12 AM

Java will further enhance platform independence through cloud-native applications, multi-platform deployment and cross-language interoperability. 1) Cloud native applications will use GraalVM and Quarkus to increase startup speed. 2) Java will be extended to embedded devices, mobile devices and quantum computers. 3) Through GraalVM, Java will seamlessly integrate with languages ​​such as Python and JavaScript to enhance cross-language interoperability.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

SAP NetWeaver Server Adapter for Eclipse

SAP NetWeaver Server Adapter for Eclipse

Integrate Eclipse with SAP NetWeaver application server.

Atom editor mac version download

Atom editor mac version download

The most popular open source editor

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function

SublimeText3 English version

SublimeText3 English version

Recommended: Win version, supports code prompts!

MantisBT

MantisBT

Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.