what is apache hadoop

Apache Hadoop is a framework for running applications on large clusters built on general-purpose hardware. It implements the Map/Reduce programming paradigm, where computing tasks are divided into small chunks (multiple times) and run on different nodes.

In addition, it also provides a distributed file system (HDFS), data is stored on computing nodes to provide extremely high cross- Data center aggregate bandwidth.

Framework role

New choice for Apache Hadoop big data ownership

Physical DAS is still the best storage for Apache Hadoop Media, because the relevant high-level professional and business companies have determined the storage media through research and practice. However, there are big problems with Apache Hadoop data storage based on HDFS.

First, the default solution is for all Apache Hadoop data to be copied, moved, and then backed up. HDFS is based on I/O optimization of Apache Hadoop large data blocks, which saves the time of Apache Hadoop data interaction. Later use usually means copying the Apache Hadoop data out. Although there are local snapshots, they are not completely consistent or fully recoverable at the point in time.

For these and other reasons, enterprise storage vendors are smart enough to make changes to HDFS, and some geek-type big data experts are making Apache Hadoop compute take advantage of external storage. But for many enterprises, Apache Hadoop offers a good compromise: no high-maintenance storage or the need to adapt to new ways of maintaining storage, which comes at a cost.

Many Apache Hadoop vendors provide remote HDFS interfaces to Apache Hadoop clusters and are the first choice for Apache Hadoop companies with relatively large business volumes. Because they will be in isilon, any other Apache Hadoop data processing big data protection, including Apache Hadoop security and other issues. Another benefit is that data stored externally can often be accessed from other Apache Hadoop protocol stores, supporting workflows and limiting the transfer of data and copies of data as needed within the enterprise. Apache Hadoop also processes big data based on this principle, a big data reference architecture, combined with a combined storage solution, directly into the Apache Hadoop cluster.

Also worth mentioning is virtualized Apache Hadoop big data analysis. In theory, all compute and storage nodes can be virtualized. VMware and RedHat/OpenStack have virtualization solutions for Hadoop. However, almost all Apache Hadoop host nodes cannot solve enterprise storage problems. It emulates the computing aspects of Apache Hadoop, allowing enterprises to accelerate and dump existing data sets - SAN/NAS - onto its HDFS overlay with Apache Hadoop. In this way, Apache Hadoop big data analysis can do no changes to the data in a data center, thereby using the new Apache Hadoop storage architecture and new data flows or any changes in data management.

Most Apache Hadoop distributions start with Apache Hadoop's open source HDFS (the current software-defined storage for big data). The difference is that Apache Hadoop takes a different approach. This is basically the storage that enterprise Apache Hadoop needs to build its own compatible storage layer on top of Apache Hadoop HDFS. The MAPR version is fully capable of handling I/O support for snapshot replication, and Apache Hadoop is also compatible with other natively supported protocols, such as NFS. Apache Hadoop is also very effective and helps in providing primarily enterprise business intelligence applications that run decision support solutions that rely on big data for historical and real-time information. Similar to the idea, IBM has released the High Performance Computing System Storage API for the Apache Hadoop distribution as an alternative to HDFS

Another interesting solution for Apache Hadoop that can help solve data problems. One is dataguise, a data security startup that can effectively protect some unique IP of Apache Hadoop's large data sets. Apache Hadoop can automatically identify and globally cover or encrypt sensitive information in a large data cluster. Horizontal data science is an emerging technology in this field. If you connect your data files to Apache Hadoop, no matter where the data is, even HDFS, Apache Hadoop will automatically store it. The output provided by Apache Hadoop big data helps to quickly build business applications, using the source and location of the data to collect the information required by the business.

If you have always held an interest in Apache Hadoop management or enterprise data center storage, this is a good time to update your knowledge of Apache Hadoop big data and if you want to keep up with Apache Hadoop big data. If you follow the footsteps, you should not refuse the application of new technologies of Apache Hadoop.

For more Apache related technical articles, please visit the Apache usage tutorial column to learn!

The above is the detailed content of what is apache hadoop. For more information, please follow other related articles on the PHP Chinese website!