[FLINK-37608][fs-connector] File sink supports to close buckets in parallel when snapshot #26498
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rallel when snapshot
davidradl
reviewed
Apr 22, 2025
| @@ -85,6 +92,8 @@ public class Buckets<IN, BucketID> { | |||
|
|
|||
| private final BucketStateSerializer<BucketID> bucketStateSerializer; | |||
|
|
|||
| private final ExecutorService snapshotActiveBucketsThreadPool; | |||
There was a problem hiding this comment.
some comments:
- I am not convinced that this is covered by existing tests as there is a new thread pool that has been introduced.
- Also I would like to see a test showing the expected behaviour when the CompletionException is thrown. I am unsure what we are looking to do when there is a CompletionException is thrown. Does everything get cleaned up when this exception is thrown?
- the title talks of close buckets but the code is in snapshotActiveBuckets. It would be easier to understand if I could see an obvious connection between close buckets and snapshotActiveBuckets.
- I see the Jira talks about introducing an option for this - but the code has not done this - it has changed it from synchronous to asynchronous. what is the thinking here?
- the CI has failed - I am not sure if it relates to this change.
|
@flinkbot run azure |
|
@flinkbot run azure |
masteryhx
reviewed
Apr 25, 2025
| } | ||
| return bucketState; | ||
| } catch (IOException e) { | ||
| throw new CompletionException(e); |
There was a problem hiding this comment.
Add some context for any exception like subtask, bucket?
| import java.util.concurrent.CompletionException; | ||
| import java.util.concurrent.ExecutorService; | ||
| import java.util.concurrent.Executors; | ||
| import java.util.stream.Collectors; | ||
|
|
||
| /** | ||
| * The manager of the different active buckets in the {@link StreamingFileSink}. |
There was a problem hiding this comment.
How about also supporting FileSink, the refactored one ?
| final BucketState<BucketID> bucketState = bucket.onReceptionOfCheckpoint(checkpointId); | ||
|
|
||
| long start = System.currentTimeMillis(); | ||
| List<CompletableFuture<BucketState<BucketID>>> futures = |
There was a problem hiding this comment.
How about adding a test about partial active buckets failure ?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Brief change log
Verifying this change
This change is already covered by existing tests, such as
BucketsTest#testSnapshotAndRestore.Does this pull request potentially affect one of the following parts:
@Public(Evolving): (yes / no)Documentation