Powered by Flink

Upcoming Events

Latest Blog Posts

A Rundown of Batch Execution Mode in the DataStream API

Apache Flink 1.12 introduced batch execution mode through FLIP-140. This blog post discusses when to use the new mode, discusses its benefits and gives examples of programs that can be run in it.

Apache Flink 1.12.2 Released

The Apache Flink community released the next bugfix version of the Apache Flink 1.12 series.

This release includes 83 fixes and minor improvements for Flink 1.12.1. The list below includes a detailed list of all fixes and improvements.

We highly recommend all users to upgrade to Flink 1.12.2.

Updated Maven dependencies:

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-java</artifactId>
  <version>1.12.2</version>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-streaming-java_2.11</artifactId>
  <version>1.12.2</version>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-clients_2.11</artifactId>
  <version>1.12.2</version>
</dependency>

You can find the binaries on the updated Downloads page.

List of resolved issues:

Sub-task

• [FLINK-21070] - Overloaded aggregate functions cause converter errors
• [FLINK-21486] - Add sanity check when switching from Rocks to Heap timers

Bug

• [FLINK-12461] - Document binary compatibility situation with Scala beyond 2.12.8
• [FLINK-16443] - Fix wrong fix for user-code CheckpointExceptions
• [FLINK-19771] - NullPointerException when accessing null array from postgres in JDBC Connector
• [FLINK-20309] - UnalignedCheckpointTestBase.execute is failed
• [FLINK-20462] - MailboxOperatorTest.testAvoidTaskStarvation
• [FLINK-20500] - UpsertKafkaTableITCase.testTemporalJoin test failed
• [FLINK-20565] - Fix typo in EXPLAIN Statements docs.
• [FLINK-20580] - Missing null value handling for SerializedValue's getByteArray()
• [FLINK-20654] - Unaligned checkpoint recovery may lead to corrupted data stream
• [FLINK-20663] - Managed memory may not be released in time when operators use managed memory frequently
• [FLINK-20675] - Asynchronous checkpoint failure would not fail the job anymore
• [FLINK-20680] - Fails to call var-arg function with no parameters
• [FLINK-20798] - Using PVC as high-availability.storageDir could not work
• [FLINK-20832] - Deliver bootstrap resouces ourselves for website and documentation
• [FLINK-20848] - Kafka consumer ID is not specified correctly in new KafkaSource
• [FLINK-20913] - Improve new HiveConf(jobConf, HiveConf.class)
• [FLINK-20921] - Fix Date/Time/Timestamp in Python DataStream
• [FLINK-20933] - Config Python Operator Use Managed Memory In Python DataStream
• [FLINK-20942] - Digest of FLOAT literals throws UnsupportedOperationException
• [FLINK-20944] - Launching in application mode requesting a ClusterIP rest service type results in an Exception
• [FLINK-20947] - Idle source doesn't work when pushing watermark into the source
• [FLINK-20961] - Flink throws NullPointerException for tables created from DataStream with no assigned timestamps and watermarks
• [FLINK-20992] - Checkpoint cleanup can kill JobMaster
• [FLINK-20998] - flink-raw-1.12.jar does not exist
• [FLINK-21009] - Can not disable certain options in Elasticsearch 7 connector
• [FLINK-21013] - Blink planner does not ingest timestamp into StreamRecord
• [FLINK-21024] - Dynamic properties get exposed to job's main method if user parameters are passed
• [FLINK-21028] - Streaming application didn't stop properly
• [FLINK-21030] - Broken job restart for job with disjoint graph
• [FLINK-21059] - KafkaSourceEnumerator does not honor consumer properties
• [FLINK-21069] - Configuration "parallelism.default" doesn't take effect for TableEnvironment#explainSql
• [FLINK-21071] - Snapshot branches running against flink-docker dev-master branch
• [FLINK-21104] - UnalignedCheckpointITCase.execute failed with "IllegalStateException"
• [FLINK-21132] - BoundedOneInput.endInput is called when taking synchronous savepoint
• [FLINK-21138] - KvStateServerHandler is not invoked with user code classloader
• [FLINK-21140] - Extract zip file dependencies before adding to PYTHONPATH
• [FLINK-21144] - KubernetesResourceManagerDriver#tryResetPodCreationCoolDown causes fatal error
• [FLINK-21155] - FileSourceTextLinesITCase.testBoundedTextFileSourceWithTaskManagerFailover does not pass
• [FLINK-21158] - wrong jvm metaspace and overhead size show in taskmanager metric page
• [FLINK-21163] - Python dependencies specified via CLI should not override the dependencies specified in configuration
• [FLINK-21169] - Kafka flink-connector-base dependency should be scope compile
• [FLINK-21208] - pyarrow exception when using window with pandas udaf
• [FLINK-21213] - e2e test fail with 'As task is already not running, no longer decline checkpoint'
• [FLINK-21215] - Checkpoint was declined because one input stream is finished
• [FLINK-21216] - StreamPandasConversionTests Fails
• [FLINK-21225] - OverConvertRule does not consider distinct
• [FLINK-21226] - Reintroduce TableColumn.of for backwards compatibility
• [FLINK-21274] - At per-job mode, during the exit of the JobManager process, if ioExecutor exits at the end, the System.exit() method will not be executed.
• [FLINK-21277] - SQLClientSchemaRegistryITCase fails to download testcontainers/ryuk:0.3.0
• [FLINK-21312] - SavepointITCase.testStopSavepointWithBoundedInputConcurrently is unstable
• [FLINK-21323] - Stop-with-savepoint is not supported by SourceOperatorStreamTask
• [FLINK-21351] - Incremental checkpoint data would be lost once a non-stop savepoint completed
• [FLINK-21361] - FlinkRelMdUniqueKeys matches on AbstractCatalogTable instead of CatalogTable
• [FLINK-21412] - pyflink DataTypes.DECIMAL is not available
• [FLINK-21452] - FLIP-27 sources cannot reliably downscale
• [FLINK-21453] - BoundedOneInput.endInput is NOT called when doing stop with savepoint WITH drain
• [FLINK-21490] - UnalignedCheckpointITCase fails on azure
• [FLINK-21492] - ActiveResourceManager swallows exception stack trace

New Feature

• [FLINK-20359] - Support adding Owner Reference to Job Manager in native kubernetes setup

Improvement

• [FLINK-9844] - PackagedProgram does not close URLClassLoader
• [FLINK-20417] - Handle "Too old resource version" exception in Kubernetes watch more gracefully
• [FLINK-20491] - Support Broadcast Operation in BATCH execution mode
• [FLINK-20517] - Support mixed keyed/non-keyed operations in BATCH execution mode
• [FLINK-20770] - Incorrect description for config option kubernetes.rest-service.exposed.type
• [FLINK-20907] - Table API documentation promotes deprecated syntax
• [FLINK-21020] - Bump Jackson to 20.10.5[.1] / 2.12.1
• [FLINK-21034] - Rework jemalloc switch to use an environment variable
• [FLINK-21035] - Deduplicate copy_plugins_if_required calls
• [FLINK-21036] - Consider removing automatic configuration fo number of slots from docker
• [FLINK-21037] - Deduplicate configuration logic in docker entrypoint
• [FLINK-21042] - Fix code example in "Aggregate Functions" section in Table UDF page
• [FLINK-21048] - Refactor documentation related to switch memory allocator
• [FLINK-21123] - Upgrade Beanutils 1.9.x to 1.9.4
• [FLINK-21164] - Jar handlers don't cleanup temporarily extracted jars
• [FLINK-21210] - ApplicationClusterEntryPoints should explicitly close PackagedProgram
• [FLINK-21381] - Kubernetes HA documentation does not state required service account and role

Task

• [FLINK-20529] - Publish Dockerfiles for release 1.12.0
• [FLINK-20534] - Add Flink 1.12 MigrationVersion
• [FLINK-20536] - Update migration tests in master to cover migration from release-1.12
• [FLINK-20960] - Add warning in 1.12 release notes about potential corrupt data stream with unaligned checkpoint
• [FLINK-21358] - Missing snapshot version compatibility for 1.12

How to natively deploy Flink on Kubernetes with High-Availability (HA)

Kubernetes provides built-in functionalities that Flink can leverage for JobManager failover. In Flink 1.12 (FLIP-144), the community implemented a Kubernetes High Availability (HA) service as an alternative to ZooKeeper for highly available production setups. In this blogpost, we will have a close look at how to deploy Flink applications natively on Kubernetes cluster with HA.

Apache Flink 1.10.3 Released

The Apache Flink community released the third bugfix version of the Apache Flink 1.10 series.

This release includes 36 fixes and minor improvements for Flink 1.10.2. The list below includes a detailed list of all fixes and improvements.

We highly recommend all users to upgrade to Flink 1.10.3.

Updated Maven dependencies:

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-java</artifactId>
  <version>1.10.3</version>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-streaming-java_2.11</artifactId>
  <version>1.10.3</version>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-clients_2.11</artifactId>
  <version>1.10.3</version>
</dependency>

You can find the binaries on the updated Downloads page.

List of resolved issues:

Bug

• [FLINK-14087] - throws java.lang.ArrayIndexOutOfBoundsException when emiting the data using RebalancePartitioner.
• [FLINK-15170] - WebFrontendITCase.testCancelYarn fails on travis
• [FLINK-15467] - Should wait for the end of the source thread during the Task cancellation
• [FLINK-16246] - Exclude "SdkMBeanRegistrySupport" from dynamically loaded AWS connectors
• [FLINK-17341] - freeSlot in TaskExecutor.closeJobManagerConnection cause ConcurrentModificationException
• [FLINK-17458] - TaskExecutorSubmissionTest#testFailingScheduleOrUpdateConsumers
• [FLINK-17677] - FLINK_LOG_PREFIX recommended in docs is not always available
• [FLINK-18081] - Fix broken links in "Kerberos Authentication Setup and Configuration" doc
• [FLINK-18196] - flink throws `NullPointerException` when executeCheckpointing
• [FLINK-18212] - Init lookup join failed when use udf on lookup table
• [FLINK-18832] - BoundedBlockingSubpartition does not work with StreamTask
• [FLINK-18959] - Fail to archiveExecutionGraph because job is not finished when dispatcher close
• [FLINK-19022] - AkkaRpcActor failed to start but no exception information
• [FLINK-19109] - Split Reader eats chained periodic watermarks
• [FLINK-19135] - (Stream)ExecutionEnvironment.execute() should not throw ExecutionException
• [FLINK-19237] - LeaderChangeClusterComponentsTest.testReelectionOfJobMaster failed with "NoResourceAvailableException: Could not allocate the required slot within slot request timeout"
• [FLINK-19401] - Job stuck in restart loop due to excessive checkpoint recoveries which block the JobMaster
• [FLINK-19557] - Issue retrieving leader after zookeeper session reconnect
• [FLINK-19675] - The plan of is incorrect when Calc contains WHERE clause, composite fields access and Python UDF at the same time
• [FLINK-19901] - Unable to exclude metrics variables for the last metrics reporter.
• [FLINK-20013] - BoundedBlockingSubpartition may leak network buffer if task is failed or canceled
• [FLINK-20018] - pipeline.cached-files option cannot escape ':' in path
• [FLINK-20033] - Job fails when stopping JobMaster
• [FLINK-20065] - UnalignedCheckpointCompatibilityITCase.test failed with AskTimeoutException
• [FLINK-20076] - DispatcherTest.testOnRemovedJobGraphDoesNotCleanUpHAFiles does not test the desired functionality
• [FLINK-20183] - Fix the default PYTHONPATH is overwritten in client side
• [FLINK-20218] - AttributeError: module 'urllib' has no attribute 'parse'
• [FLINK-20875] - [CVE-2020-17518] Directory traversal attack: remote file writing through the REST API

Improvement

• [FLINK-16753] - Exception from AsyncCheckpointRunnable should be wrapped in CheckpointException
• [FLINK-18287] - Correct the documentation of Python Table API in SQL pages
• [FLINK-19055] - MemoryManagerSharedResourcesTest contains three tests running extraordinary long
• [FLINK-19105] - Table API Sample Code Error
• [FLINK-19252] - Jaas file created under io.tmp.dirs - folder not created if not exists
• [FLINK-19339] - Support Avro's unions with logical types
• [FLINK-19523] - Hide sensitive command-line configurations

Task

• [FLINK-20906] - Update copyright year to 2021 for NOTICE files

Apache Flink 1.12.1 Released

The Apache Flink community released the first bugfix version of the Apache Flink 1.12 series.

This release includes 79 fixes and minor improvements for Flink 1.12.0. The list below includes a detailed list of all fixes and improvements.

We highly recommend all users to upgrade to Flink 1.12.1.

Attention: Using unaligned checkpoints in Flink 1.12.0 combined with two/multiple inputs tasks or with union inputs for single input tasks can result in corrupted state.

This can happen if a new checkpoint is triggered before recovery is fully completed. For state to be corrupted a task with two or more input gates must receive a checkpoint barrier exactly at the same time this tasks finishes recovering spilled in-flight data. In such case this new checkpoint can succeed, with corrupted/missing in-flight data, which will result in various deserialisation/corrupted data stream errors when someone attempts to recover from such corrupted checkpoint.

Using unaligned checkpoints in Flink 1.12.1, a corruption may occur in the checkpoint following a declined checkpoint.

A late barrier of a canceled checkpoint may lead to buffers being not written into the successive checkpoint, such that recovery is not possible. This happens, when the next checkpoint barrier arrives at a given operator before all previous barriers arrived, which can only happen after cancellation in unaligned checkpoints.

Updated Maven dependencies:

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-java</artifactId>
  <version>1.12.1</version>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-streaming-java_2.11</artifactId>
  <version>1.12.1</version>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-clients_2.11</artifactId>
  <version>1.12.1</version>
</dependency>

You can find the binaries on the updated Downloads page.

List of resolved issues:

Sub-task

• [FLINK-18897] - Add documentation for the maxwell-json format
• [FLINK-20352] - Rework command line interface documentation page
• [FLINK-20353] - Rework logging documentation page
• [FLINK-20354] - Rework standalone deployment documentation page
• [FLINK-20355] - Rework K8s deployment documentation page
• [FLINK-20356] - Rework Mesos deployment documentation page
• [FLINK-20422] - Remove from .html files in flink documentation
• [FLINK-20485] - Map views are deserialized multiple times
• [FLINK-20601] - Rework PyFlink CLI documentation

Bug

• [FLINK-19369] - BlobClientTest.testGetFailsDuringStreamingForJobPermanentBlob hangs
• [FLINK-19435] - Deadlock when loading different driver classes concurrently using Class.forName
• [FLINK-19725] - Logger cannot be initialized due to timeout: LoggerInitializationException is thrown
• [FLINK-19880] - Fix ignore-parse-errors not work for the legacy JSON format
• [FLINK-20213] - Partition commit is delayed when records keep coming
• [FLINK-20221] - DelimitedInputFormat does not restore compressed filesplits correctly leading to dataloss
• [FLINK-20273] - Fix Table api Kafka connector Sink Partitioner Document Error
• [FLINK-20372] - Update Kafka SQL connector page to mention properties.* options
• [FLINK-20389] - UnalignedCheckpointITCase failure caused by NullPointerException
• [FLINK-20404] - ZooKeeper quorum fails to start due to missing log4j library
• [FLINK-20419] - Insert fails due to failure to generate execution plan
• [FLINK-20428] - ZooKeeperLeaderElectionConnectionHandlingTest.testConnectionSuspendedHandlingDuringInitialization failed with "No result is expected since there was no leader elected before stopping the server, yet"
• [FLINK-20429] - KafkaTableITCase.testKafkaTemporalJoinChangelog failed with unexpected results
• [FLINK-20433] - UnalignedCheckpointTestBase.execute failed with "TestTimedOutException: test timed out after 300 seconds"
• [FLINK-20464] - Some Table examples are not built correctly
• [FLINK-20467] - Fix the Example in Python DataStream Doc
• [FLINK-20470] - MissingNode can't be casted to ObjectNode when deserializing JSON
• [FLINK-20476] - New File Sink end-to-end test Failed
• [FLINK-20486] - Hive temporal join should allow monitor interval smaller than 1 hour
• [FLINK-20492] - The SourceOperatorStreamTask should implement cancelTask() and finishTask()
• [FLINK-20493] - SQLClientSchemaRegistryITCase failed with "Could not build the flink-dist image"
• [FLINK-20521] - Null result values are being swallowed by RPC system
• [FLINK-20525] - StreamArrowPythonGroupWindowAggregateFunctionOperator doesn't handle rowtime and proctime properly
• [FLINK-20543] - Fix typo in upsert kafka docs
• [FLINK-20554] - The Checkpointed Data Size of the Latest Completed Checkpoint is incorrectly displayed on the Overview page of the UI
• [FLINK-20582] - Fix typos in `CREATE Statements` docs
• [FLINK-20607] - a wrong example in udfs page.
• [FLINK-20615] - Local recovery and sticky scheduling end-to-end test timeout with "IOException: Stream Closed"
• [FLINK-20626] - Canceling a job when it is failing will result in job hanging in CANCELING state
• [FLINK-20630] - [Kinesis][DynamoDB] DynamoDB Streams Consumer fails to consume from Latest
• [FLINK-20646] - ReduceTransformation does not work with RocksDBStateBackend
• [FLINK-20648] - Unable to restore job from savepoint when using Kubernetes based HA services
• [FLINK-20664] - Support setting service account for TaskManager pod
• [FLINK-20665] - FileNotFoundException when restore from latest Checkpoint
• [FLINK-20666] - Fix the deserialized Row losing the field_name information in PyFlink
• [FLINK-20669] - Add the jzlib LICENSE file in flink-python module
• [FLINK-20703] - HiveSinkCompactionITCase test timeout
• [FLINK-20704] - Some rel data type does not implement the digest correctly
• [FLINK-20756] - PythonCalcSplitConditionRule is not working as expected
• [FLINK-20764] - BatchGroupedReduceOperator does not emit results for singleton inputs
• [FLINK-20781] - UnalignedCheckpointITCase failure caused by NullPointerException
• [FLINK-20784] - .staging_xxx does not exist, when insert into hive
• [FLINK-20793] - Fix NamesTest due to code style refactor
• [FLINK-20803] - Version mismatch between spotless-maven-plugin and google-java-format plugin
• [FLINK-20841] - Fix compile error due to duplicated generated files

Improvement

• [FLINK-19013] - Log start/end of state restoration
• [FLINK-19259] - Use classloader release hooks with Kinesis producer to avoid metaspace leak
• [FLINK-19832] - Improve handling of immediately failed physical slot in SlotSharingExecutionSlotAllocator
• [FLINK-20055] - Datadog API Key exposed in Flink JobManager logs
• [FLINK-20168] - Translate page 'Flink Architecture' into Chinese
• [FLINK-20209] - Add missing checkpoint configuration to Flink UI
• [FLINK-20298] - Replace usage of in flink documentation
• [FLINK-20468] - Enable leadership control in MiniCluster to test JM failover
• [FLINK-20510] - Enable log4j2 monitor interval by default
• [FLINK-20519] - Extend HBase notice with transitively bundled dependencies
• [FLINK-20570] - The `NOTE` tip style is different from the others in process_function page.
• [FLINK-20588] - Add docker-compose as appendix to Mesos documentation
• [FLINK-20629] - [Kinesis][EFO] Migrate from DescribeStream to DescribeStreamSummary
• [FLINK-20647] - Use yield to generate output datas in ProcessFunction for Python DataStream
• [FLINK-20650] - Mark "native-k8s" as deprecated in docker-entrypoint.sh
• [FLINK-20651] - Use Spotless/google-java-format for code formatting/enforcement
• [FLINK-20682] - Add configuration options related to hadoop
• [FLINK-20697] - Correct the Type of "lookup.cache.ttl" in jdbc.md/jdbc.zh.md
• [FLINK-20790] - Generated classes should not be put under src/ directory
• [FLINK-20792] - Allow shorthand invocation of spotless
• [FLINK-20805] - Blink runtime classes partially ignored by spotless
• [FLINK-20822] - Don't check whether a function is generic in hive catalog
• [FLINK-20866] - Add how to list jobs in Yarn deployment documentation when HA enabled

Task

• [FLINK-20300] - Create Flink 1.12 release notes
• [FLINK-20906] - Update copyright year to 2021 for NOTICE files