This article mainly introduces relevant information about the detailed explanation of custom OutputFormat examples in java. Here are examples to help you learn and understand this part of the content. I hope this article can help everyone. Friends in need can refer to it
Detailed explanation of examples of custom OutputFormat in java
Example code:
package com.ccse.hadoop.outputformat; import java.io.IOException; import java.net.URI; import java.net.URISyntaxException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.JobContext; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.OutputCommitter; import org.apache.hadoop.mapreduce.OutputFormat; import org.apache.hadoop.mapreduce.RecordWriter; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.TaskAttemptContext; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter; public class MySelfOutputFormatApp { public final static String INPUT_PATH = "hdfs://chaoren1:9000/mapinput"; public final static String OUTPUT_PATH = "hdfs://chaoren1:9000/mapoutput"; public final static String OUTPUT_FILENAME = "/abc"; public static void main(String[] args) throws IOException, URISyntaxException, ClassNotFoundException, InterruptedException { Configuration conf = new Configuration(); FileSystem fileSystem = FileSystem.get(new URI(OUTPUT_PATH), conf); fileSystem.delete(new Path(OUTPUT_PATH), true); Job job = new Job(conf, MySelfOutputFormatApp.class.getSimpleName()); job.setJarByClass(MySelfOutputFormatApp.class); FileInputFormat.setInputPaths(job, new Path(INPUT_PATH)); job.setMapperClass(MyMapper.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(LongWritable.class); job.setReducerClass(MyReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(LongWritable.class); job.setOutputFormatClass(MyselfOutputFormat.class); job.waitForCompletion(true); } public static class MyMapper extends Mapper<LongWritable, Text, Text, LongWritable> { private Text word = new Text(); private LongWritable writable = new LongWritable(1); @Override protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, LongWritable>.Context context) throws IOException, InterruptedException { if (value != null) { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, writable); } } } } public static class MyReducer extends Reducer<Text, LongWritable, Text, LongWritable> { @Override protected void reduce(Text key, Iterable<LongWritable> values, Reducer<Text, LongWritable, Text, LongWritable>.Context context) throws IOException, InterruptedException { long sum = 0; for (LongWritable value : values) { sum += value.get(); } context.write(key, new LongWritable(sum)); } } public static class MyselfOutputFormat extends OutputFormat<Text, LongWritable> { private FSDataOutputStream outputStream = null; @Override public RecordWriter<Text, LongWritable> getRecordWriter( TaskAttemptContext context) throws IOException, InterruptedException { try { FileSystem fileSystem = FileSystem.get(new URI(MySelfOutputFormatApp.OUTPUT_PATH), context.getConfiguration()); //指定文件的输出路径 final Path path = new Path(MySelfOutputFormatApp.OUTPUT_PATH + MySelfOutputFormatApp.OUTPUT_FILENAME); this.outputStream = fileSystem.create(path, false); } catch (URISyntaxException e) { e.printStackTrace(); } return new MySelfRecordWriter(outputStream); } @Override public void checkOutputSpecs(JobContext context) throws IOException, InterruptedException { } @Override public OutputCommitter getOutputCommitter(TaskAttemptContext context) throws IOException, InterruptedException { return new FileOutputCommitter(new Path(MySelfOutputFormatApp.OUTPUT_PATH), context); } } public static class MySelfRecordWriter extends RecordWriter<Text, LongWritable> { private FSDataOutputStream outputStream = null; public MySelfRecordWriter(FSDataOutputStream outputStream) { this.outputStream = outputStream; } @Override public void write(Text key, LongWritable value) throws IOException, InterruptedException { this.outputStream.writeBytes(key.toString()); this.outputStream.writeBytes("\t"); this.outputStream.writeLong(value.get()); } @Override public void close(TaskAttemptContext context) throws IOException, InterruptedException { this.outputStream.close(); } } }
2.OutputFormat is used Handles various output destinations.
2.1 The key-value pairs that OutputFormat needs to write out come from the Reducer class and are obtained through RecordWriter.
2.2 The write(...) method in RecordWriter only has k and v. Where should it be written? This is handled by passing in the OutputStream separately. write is to write k and v into the OutputStream.
2.3 RecordWriter class is located in OutputFormat. Therefore, our custom OutputFromat must inherit the OutputFormat type. Then, the stream object must be obtained in the getRecordWriter(...) method.
The above is the detailed content of Specific analysis of Java custom OutputFormat. For more information, please follow other related articles on the PHP Chinese website!

Javadevelopmentisnotentirelyplatform-independentduetoseveralfactors.1)JVMvariationsaffectperformanceandbehavioracrossdifferentOS.2)NativelibrariesviaJNIintroduceplatform-specificissues.3)Filepathsandsystempropertiesdifferbetweenplatforms.4)GUIapplica

Java code will have performance differences when running on different platforms. 1) The implementation and optimization strategies of JVM are different, such as OracleJDK and OpenJDK. 2) The characteristics of the operating system, such as memory management and thread scheduling, will also affect performance. 3) Performance can be improved by selecting the appropriate JVM, adjusting JVM parameters and code optimization.

Java'splatformindependencehaslimitationsincludingperformanceoverhead,versioncompatibilityissues,challengeswithnativelibraryintegration,platform-specificfeatures,andJVMinstallation/maintenance.Thesefactorscomplicatethe"writeonce,runanywhere"

Platformindependenceallowsprogramstorunonanyplatformwithoutmodification,whilecross-platformdevelopmentrequiressomeplatform-specificadjustments.Platformindependence,exemplifiedbyJava,enablesuniversalexecutionbutmaycompromiseperformance.Cross-platformd

JITcompilationinJavaenhancesperformancewhilemaintainingplatformindependence.1)Itdynamicallytranslatesbytecodeintonativemachinecodeatruntime,optimizingfrequentlyusedcode.2)TheJVMremainsplatform-independent,allowingthesameJavaapplicationtorunondifferen

Javaispopularforcross-platformdesktopapplicationsduetoits"WriteOnce,RunAnywhere"philosophy.1)ItusesbytecodethatrunsonanyJVM-equippedplatform.2)LibrarieslikeSwingandJavaFXhelpcreatenative-lookingUIs.3)Itsextensivestandardlibrarysupportscompr

Reasons for writing platform-specific code in Java include access to specific operating system features, interacting with specific hardware, and optimizing performance. 1) Use JNA or JNI to access the Windows registry; 2) Interact with Linux-specific hardware drivers through JNI; 3) Use Metal to optimize gaming performance on macOS through JNI. Nevertheless, writing platform-specific code can affect the portability of the code, increase complexity, and potentially pose performance overhead and security risks.

Java will further enhance platform independence through cloud-native applications, multi-platform deployment and cross-language interoperability. 1) Cloud native applications will use GraalVM and Quarkus to increase startup speed. 2) Java will be extended to embedded devices, mobile devices and quantum computers. 3) Through GraalVM, Java will seamlessly integrate with languages such as Python and JavaScript to enhance cross-language interoperability.


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

SAP NetWeaver Server Adapter for Eclipse
Integrate Eclipse with SAP NetWeaver application server.

Atom editor mac version download
The most popular open source editor

EditPlus Chinese cracked version
Small size, syntax highlighting, does not support code prompt function

SublimeText3 English version
Recommended: Win version, supports code prompts!

MantisBT
Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.
