What is apache flink

Apache Flink is an open source stream processing framework developed by the Apache Software Foundation. Its core is a distributed streaming data flow engine written in Java and Scala. Flink executes arbitrary streaming data programs in a data parallel and pipeline manner. Flink's pipeline runtime system can execute batch and stream processing programs. In addition, Flink’s runtime itself also supports the execution of iterative algorithms.

Flink provides a high-throughput, low-latency streaming data engine and support for event-time processing and state management. Flink applications are fault-tolerant in the event of machine failures and support exactly-once semantics. Programs can be written in languages ​​such as Java, Scala [9], Python [9] and SQL, and are automatically compiled and optimized into data flow programs that run in a cluster or cloud environment.

Flink does not provide its own data storage system, but provides data sources and sinks for systems such as Amazon Kinesis, Apache Kafka, HDFS, Apache Cassandra and ElasticSearch.

Apache Flink’s data flow programming model provides event-at-a-time processing on finite and infinite data sets. At a basic level, Flink programs consist of streams and transformations. "Conceptually, a stream is a (possibly never-ending) record of a stream of data, and a transformation is an operation that takes one or more streams as input and therefore produces one or more output streams".

Apache Flink includes two core APIs: the Data Stream API for bounded or unbounded data streams and the Data Set API for bounded data sets. Flink also provides a table API, a SQL-like expression language for relational streaming and batch processing that can be easily embedded into Flink’s dataflow and dataset APIs. The highest-level language supported by Flink is SQL, which is semantically similar to the table API and represents programs as SQL query expressions.

For more Apache related knowledge, please visit the Apache usage tutorial column!

The above is the detailed content of What is apache flink. For more information, please follow other related articles on the PHP Chinese website!