Java developers must know these 15 big data tools and frameworks

A framework is actually a collection of reusable codes. The code of the framework is the code of the framework architecture, not the business logic code. The framework code protects classes, methods, functions, etc. The framework code is combined according to certain rules to form a framework.

Programmers at most large domestic Internet companies are called R&D engineers, but in fact there are almost no R&D projects in the country and can only be called development.

Most of the work of development programmers is repetitive work, which is easy to cause fatigue. The salary reaches a peak within 2-5 years of work, and it is more difficult to improve. This has led to many programmers ending up Switched to other industries.

JAVA is sophisticated and powerful, and has irreplaceable performance and maintainability of other languages. It has already become one of the most popular programming languages. Many people want to enter the IT industry, and the language of choice is JAVA.

However, in the next 10 years, the world will definitely be dominated by big data. With the outbreak of artificial intelligence, a large number of companies will enter the field of big data, and those who switch from JAVA programmers will JAVA big data will have natural advantages, because the current big data architecture is basically completed in the JAVA language. In the next 10 years, the demand for JAVA big data will be increasing.

If those who are learning JAVA now do not want to be eliminated in the future, they will definitely join the ranks of big data in the future. According to the current industry trends, JAVA programmers are less competitive due to development limitations and age. It is also becoming more and more easy to be overtaken by the younger generation in terms of technology. Because JAVA programmers work too long overtime, a large number of senior JAVA engineers have already entered big data.

The biggest challenge faced by IT developers today is complexity. Hardware is becoming more and more complex, OS is becoming more and more complex, programming languages ​​and APIs are becoming more and more complex, and the applications we build are also becoming more and more complex. According to a survey report by foreign media, experts have listed some tools or frameworks that Java programmers have been using in the past 12 months, which may be meaningful to you.

1. MongoDB - the most popular, cross-platform, document-oriented database.

MongoDB is a database based on distributed file storage, written in C language. Designed to provide scalable, high-performance data storage solutions for web applications. Application performance depends on database performance. MongoDB is the most feature-rich among non-relational databases and is most like a relational database. With the release of MongoDB 3.4, its application scenario capabilities have been further expanded.

The core advantages of MongoDB are its flexible document model, highly available replica sets, and scalable sharded clusters. You can try to understand MongoDB from several aspects, such as real-time monitoring of MongoDB tools, memory usage and page faults, number of connections, database operations, replication sets, etc.

2. Elasticsearch - a distributed RESTful search engine built for the cloud.

ElasticSearch is a search server based on Lucene. It provides a distributed multi-user capable full-text search engine based on a RESTful web interface. Elasticsearch, developed in Java and released as open source under the terms of the Apache license, is a popular enterprise-level search engine.

ElasticSearch is not only a full-text search engine, but also a distributed real-time document storage, in which each field is indexed data and can be searched; it is also a distributed search engine with real-time analysis capabilities. And it can be expanded to hundreds of servers to store and process petabytes of data. ElasticSearch uses Lucene to complete its indexing function at the bottom level, so many of its basic concepts originate from Lucene.

3. Cassandra - an open source distributed database management system, originally developed by Facebook, designed to handle large amounts of data on many commodity servers, provide high availability, and have no single point of failure.

Apache Cassandra is an open source distributed NoSQL database system. It combines the data model of Google BigTable with the fully distributed architecture of Amazon Dynamo. It was open sourced in 2008. Since then, due to its good scalability, Cassandra has been adopted by Web 2.0 websites such as Digg and Twitter, and has become a popular distributed structured data storage solution.

Because Cassandra is written in Java, it can theoretically run on machines with JDK6 and above. The officially tested JDKs include OpenJDK and Sun's JDK. The operation commands of Cassandra are similar to the relational databases we usually operate. For friends who are familiar with MySQL, the operation will be easy to get started.

4. Redis - open source (BSD licensed) in-memory data structure storage, used as a database, cache and message broker.

Redis is an open source log-type Key-Value database written in ANSI C language, supports network, can be memory-based and persistent, and provides APIs in multiple languages. Redis has three main features that set it apart from many other competitors: Redis is a database that saves data entirely in memory, using disk only for persistence purposes; Redis has a relatively rich set of data types compared to many key-value data storage systems ; Redis can copy data to any number

5. Hazelcast - an open source memory data grid based on Java.

Hazelcast is an in-memory data grid that provides Java programmers with mission-critical transactions and trillion-level memory applications. Although Hazelcast does not have a so-called "Master", it still has a Leader node (the oldest member). This concept is similar to the Leader in ZooKeeper, but the implementation principle is completely different. At the same time, the data in Hazelcast is distributed, and each member holds part of the data and corresponding backup data, which is also different from ZooKeeper.

The application convenience of Hazelcast is loved by developers, but if you want to put it into use, you need to consider it carefully.

6. EHCache - a widely used open source Java distributed cache. Mainly for general cache, Java EE and lightweight containers.

EhCache is a pure Java in-process caching framework, which is fast and capable. It is the default CacheProvider in hibernate. The main features are: fast and simple, with multiple caching strategies; cached data has two levels, memory and disk, so there is no need to worry about capacity issues; cached data will be written to disk during the restart of the virtual machine; it can be accessed through RMI and pluggable API Distributed caching in other ways; has a listening interface for cache and cache managers; supports multiple cache manager instances, as well as multiple cache areas for one instance; provides Hibernate cache implementation.

7. Hadoop - an open source software framework written in Java, used for distributed storage, and for very large data users can develop distributed programs without understanding the underlying details of distribution.

Make full use of the cluster for high-speed computing and storage. Hadoop implements a distributed file system (Hadoop Distributed File System), referred to as HDFS. The core design of the Hadoop framework is: HDFS and MapReduce. HDFS provides storage for massive data, and MapReduce provides calculation for massive data.

8. Solr - an open source enterprise search platform, written in Java, from the Apache Lucene project.

Solr is an independent enterprise-level search application server that provides an API interface similar to Web-service. Users can submit XML files in a certain format to the search engine server through http requests to generate indexes; they can also make search requests through Http Get operations and get returned results in XML format.

Like ElasticSearch, it is also based on Lucene, but it extends it to provide a richer query language than Lucene, while being configurable, scalable and optimizing query performance.

9. Spark - the most active project in the Apache Software Foundation, is an open source cluster computing framework.

Spark is an open source cluster computing environment similar to Hadoop, but there are some differences between the two that make Spark superior in certain workloads. In other words That being said, Spark enables in-memory distributed datasets, which in addition to being able to provide interactive queries, can also optimize iterative workloads.

Spark is implemented in the Scala language and uses Scala as its application framework. Unlike Hadoop, Spark and Scala are tightly integrated, and Scala can operate as easily as local collection objects.

10. Memcached - a general distributed memory cache system.

Memcached is a distributed caching system that was originally developed by Danga Interactive for LiveJournal, but is used by many software (such as MediaWiki). As a high-speed distributed cache server, Memcached has the following characteristics: simple protocol, event processing based on libevent, and built-in memory storage.

11. Apache Hive--Provides a SQL-like layer on top of Hadoop.

Hive is a data warehouse platform based on Hadoop. Through hive, ETL work can be easily performed. hive defines a query language similar to SQL, which can convert user-written SQL into corresponding Mapreduce programs for execution based on Hadoop. Currently, Apache Hive 2.1.1 version has been released.

12. Apache Kafka - a high-throughput, distributed subscription messaging system originally developed by LinkedIn.

Apache Kafka is an open source messaging system project written in Scala. The goal of this project is to provide a unified, high-throughput, low-latency platform for processing real-time data. Kafka maintains messages differentiated by classes, called topics. Producers publish messages to Kafka topics, and consumers register with topics and receive messages published to these topics.

13. Akka – A toolkit for building highly concurrent, distributed and resilient message-driven applications on the JVM.

Akka is a library written in Scala that simplifies writing fault-tolerant, highly scalable Java and Scala actor model applications. It has been successfully used in the telecommunications industry, and the system almost never goes down.

14. HBase - open source, non-relational, distributed database, modeled using Google's BigTable, written in Java, and runs on HDFS.

Different from commercial big data products such as FUJITSU Cliq, HBase is an open source implementation of Google Bigtable. Similar to Google Bigtable, which uses GFS as its file storage system, HBase uses Hadoop HDFS as its file storage system; Google runs MapReduce to process Bigtable. Massive data, HBase also uses Hadoop MapReduce to process massive data in HBase; Google Bigtable uses Chubby as a collaborative service, and HBase uses Zookeeper as a counterpart.

15. Neo4j - an open source graph database implemented in Java.

Neo4j is a high-performance NOSQL graph database that stores structured data on the network instead of in tables. It is an embedded, disk-based, fully transactional Java persistence engine.

Summary:

Java is sought after by millions of developers around the world and has evolved into an excellent programming language. Ultimately, the language is constantly being improved to meet changing market needs as technology changes.

Whether you own a technology company or not, software has become an integral part of almost every business, and in order to attract your potential customers, you should deliver a technically innovative product to your customers. Well, Java can provide such a platform to help you realize this technological innovation. Java contributors have been maintaining extensive updates to provide the latest and most powerful features.

Related recommendations:

ThinkPHP framework one, ThinkPHP framework

ci framework (1), ci framework (

The above is the detailed content of Java developers must know these 15 big data tools and frameworks. For more information, please follow other related articles on the PHP Chinese website!