Custom, Complex Windows at Scale Using Apache Flink Matt Zimmer - - PowerPoint PPT Presentation

▶
custom complex windows at scale using apache flink
SMART_READER_LITE
LIVE PREVIEW

Custom, Complex Windows at Scale Using Apache Flink Matt Zimmer - - PowerPoint PPT Presentation

Jan 09, 2023 263 likes •1.36k views

Custom, Complex Windows at Scale Using Apache Flink Matt Zimmer QCon San Francisco 14 November | 2017 @zimmermatt Agenda. Motivating Use Cases. Window Requirements. The Solution (Conceptual). Event Processing Flow.

slide-1
SLIDE 1

Matt Zimmer QCon San Francisco 14 November | 2017

Custom, Complex Windows at Scale Using Apache Flink

@zimmermatt

slide-2
SLIDE 2

Agenda.

  • Motivating Use Cases.
  • Window Requirements.
  • The Solution (Conceptual).
  • Event Processing Flow.
  • Apache Flink Window API Walk-Through.
  • The Solution (Detail).
  • Pitfalls to Watch Out For.
  • Alternative Implementations.
  • Questions.

@zimmermatt

slide-3
SLIDE 3
  • Motivating Use Cases.
  • Window Requirements.
  • The Solution (Conceptual).
  • Event Processing Flow.
  • Apache Flink Window API Walk-Through.
  • The Solution in (Detail).
  • Pitfalls to Watch Out For.
  • Alternative Implementations.
  • Questions.

@zimmermatt

slide-4
SLIDE 4

@zimmermatt

Time User A User B User C

slide-5
SLIDE 5

@zimmermatt

Use Case: Most Simple.

Time User A User B User C

t4 t3 t1 t2 t5 t4 t3 t1 t2 t7 t5 t6 t3 t1 t2 t5 t9 t7 t8

slide-6
SLIDE 6

@zimmermatt

Use Case: Most Simple.

Time User A User B User C

t4 t3 t1 t2 t5 t4 t3 t1 t2 t7 t5 t6 t3 t1 t2 t5 t9 t7 t8

slide-7
SLIDE 7

@zimmermatt

Use Case: More Complex.

Time User A User B User C

t4 t3 t1 t2 t5 t4 t3 t1 t2 t7 t5 t6 t3 t1 t2 t5 t9 t7 t8

slide-8
SLIDE 8

@zimmermatt

Use Case: More Complex.

Time User A User B User C

t4 t3 t1 t2 t5 t4 t3 t1 t2 t7 t5 t6 t3 t1 t2 t5 t9 t7 t8

slide-9
SLIDE 9

@zimmermatt

Use Case: Most Complex.

Time User A User B User C

t4 t3 t1 t2 t5 t3 t1 t2 t5

slide-10
SLIDE 10

@zimmermatt

Use Case: Most Complex.

Time User A User B User C

t4 t3 t1 t2 t5 t3 t1 t2 t5

slide-11
SLIDE 11
  • Events

○ Millions per second ○ 100s billions per day

  • Data Flowing In

○ 10s of GiB per second ○ Low (single digit) PiB per day

  • State

○ 10s of TiB

Targeted Scale.

@zimmermatt

slide-12
SLIDE 12
  • Motivating Use Cases.
  • Window Requirements.
  • The Solution (Conceptual).
  • Event Processing Flow.
  • Apache Flink Window API Walk-Through.
  • The Solution in (Detail).
  • Pitfalls to Watch Out For.
  • Alternative Implementations.
  • Questions.

@zimmermatt

slide-13
SLIDE 13
  • Unaligned windows
  • Bounded by event type
  • Handle out of order events
  • Emit early results
  • Capture relevant events; ignore the rest

Window Requirements.

@zimmermatt

slide-14
SLIDE 14

Can we use a standard window type?

@zimmermatt

slide-15
SLIDE 15

@zimmermatt

Tumbling Window?

Time User A User B User C

slide-16
SLIDE 16

@zimmermatt

Sliding Window?

Time User A User B User C

slide-17
SLIDE 17

@zimmermatt

Sliding Window?

Time User A

slide-18
SLIDE 18

@zimmermatt

Sliding Window?

Time User A

slide-19
SLIDE 19

@zimmermatt

Sliding Window?

Time User A

slide-20
SLIDE 20

@zimmermatt

Sliding Window?

Time User A

slide-21
SLIDE 21

@zimmermatt

Apache Beam Session Window?

Time User A User B User C

slide-22
SLIDE 22

@zimmermatt

Apache Beam Session Window?

Time User C

slide-23
SLIDE 23

@zimmermatt

Apache Beam Session Window?

Time User C

slide-24
SLIDE 24

@zimmermatt

Apache Beam Session Window?

Time User C

slide-25
SLIDE 25

@zimmermatt

Apache Beam Session Window?

Time User C

slide-26
SLIDE 26

@zimmermatt

Apache Beam Session Window?

Time User C

slide-27
SLIDE 27

@zimmermatt

Apache Beam Session Window?

Time User C

slide-28
SLIDE 28
  • Motivating Use Cases.
  • Window Requirements.
  • The Solution (Conceptual).
  • Event Processing Flow.
  • Apache Flink Window API Walk-Through.
  • The Solution in (Detail).
  • Pitfalls to Watch Out For.
  • Alternative Implementations.
  • Questions.

@zimmermatt

slide-29
SLIDE 29
  • Unaligned windows
  • Bounded by event type
  • Handle out of order events
  • Emit early results
  • Capture relevant events; ignore the rest

@zimmermatt

Window Requirements Redux.

slide-30
SLIDE 30

The solution

at 10,000 feet.

@zimmermatt

slide-31
SLIDE 31

@zimmermatt

The Solution (Conceptual).

Time User A User B User C

t4 t3 t1 t2 t5 t3 t1 t2 t5

slide-32
SLIDE 32

@zimmermatt

Time User A User B User C

t4 t3 t1 t2 t5 t3 t1 t2 t5

The Solution (Conceptual).

slide-33
SLIDE 33

@zimmermatt

The Solution (Conceptual).

Time User A

t4 t3 t1 t2 t5

slide-34
SLIDE 34

@zimmermatt

The Solution (Conceptual).

Time User A

t4 t3 t1 t2 t5

slide-35
SLIDE 35

@zimmermatt

The Solution (Conceptual).

Time User A

t4 t3 t1 t2 t5

slide-36
SLIDE 36

@zimmermatt

The Solution (Conceptual).

Time User A

t4 t3 t1 t2 t5

Watermark

slide-37
SLIDE 37

@zimmermatt

The Solution (Conceptual).

Time User A

t4 t3 t1 t2 t5

Watermark

slide-38
SLIDE 38

@zimmermatt

The Solution (Conceptual).

Time User A

t4 t3 t1 t2 t5

Watermark

slide-39
SLIDE 39

@zimmermatt

The Solution (Conceptual).

Time User A

t4 t3 t1 t2 t5

Watermark

slide-40
SLIDE 40

@zimmermatt

The Solution (Conceptual).

Time User A

t4 t3 t1 t2 t5

Watermark

slide-41
SLIDE 41

@zimmermatt

The Solution (Conceptual).

Time User A

t4 t3 t1 t2 t5

Watermark

slide-42
SLIDE 42

@zimmermatt

The Solution (Conceptual).

Time User A

t4 t3 t1 t2 t5

Watermark

slide-43
SLIDE 43

@zimmermatt

The Solution (Conceptual).

Time User A

t4 t3 t1 t2 t5

Watermark

slide-44
SLIDE 44

@zimmermatt

The Solution (Conceptual).

Time User A

t4 t3 t1 t2 t5

Watermark

slide-45
SLIDE 45
  • Motivating Use Cases.
  • Window Requirements.
  • The Solution (Conceptual).
  • Event Processing Flow.
  • Apache Flink Window API Walk-Through.
  • The Solution in (Detail).
  • Pitfalls to Watch Out For.
  • Alternative Implementations.
  • Questions.

@zimmermatt

slide-46
SLIDE 46

1. Window assigner.

@zimmermatt

Event processing flow.

slide-47
SLIDE 47

1. Window assigner. a. Assign Event to Window(s) (assignWindows).

@zimmermatt

Event processing flow.

slide-48
SLIDE 48

1. Window assigner. a. Assign Event to Window(s) (assignWindows). b. Merge Windows (mergeWindows).

@zimmermatt

Event processing flow.

slide-49
SLIDE 49

1. Window assigner. a. Assign Event to Window(s) (assignWindows). b. Merge Windows (mergeWindows). 2. Trigger Handlers.

@zimmermatt

Event processing flow.

slide-50
SLIDE 50

1. Window assigner. a. Assign Event to Window(s) (assignWindows). b. Merge Windows (mergeWindows). 2. Trigger Handlers. a. Element (onElement).

@zimmermatt

Event processing flow.

slide-51
SLIDE 51

1. Window assigner. a. Assign Event to Window(s) (assignWindows). b. Merge Windows (mergeWindows). 2. Trigger Handlers. a. Element (onElement). b. Merge (onMerge).

@zimmermatt

Event processing flow.

slide-52
SLIDE 52

1. Window assigner. a. Assign Event to Window(s) (assignWindows). b. Merge Windows (mergeWindows). 2. Trigger Handlers. a. Element (onElement). b. Merge (onMerge). 3. Trigger Timers.

@zimmermatt

Event processing flow.

slide-53
SLIDE 53

1. Window assigner. a. Assign Event to Window(s) (assignWindows). b. Merge Windows (mergeWindows). 2. Trigger Handlers. a. Element (onElement). b. Merge (onMerge). 3. Trigger Timers. a. Processing Time (onProcessingTime).

@zimmermatt

Event processing flow.

slide-54
SLIDE 54

1. Window assigner. a. Assign Event to Window(s) (assignWindows). b. Merge Windows (mergeWindows). 2. Trigger Handlers. a. Element (onElement). b. Merge (onMerge). 3. Trigger Timers. a. Processing Time (onProcessingTime). b. Event Time (onEventTime).

@zimmermatt

Event processing flow.

slide-55
SLIDE 55

1. Window assigner. a. Assign Event to Window(s) (assignWindows). b. Merge Windows (mergeWindows). 2. Trigger Handlers. a. Element (onElement). b. Merge (onMerge). 3. Trigger Timers. a. Processing Time (onProcessingTime). b. Event Time (onEventTime). 4. Evictor.

@zimmermatt

Event processing flow.

slide-56
SLIDE 56

1. Window assigner. a. Assign Event to Window(s) (assignWindows). b. Merge Windows (mergeWindows). 2. Trigger Handlers. a. Element (onElement). b. Merge (onMerge). 3. Trigger Timers. a. Processing Time (onProcessingTime). b. Event Time (onEventTime). 4. Evictor. a. Before (evictBefore).

@zimmermatt

Event processing flow.

slide-57
SLIDE 57

1. Window assigner. a. Assign Event to Window(s) (assignWindows). b. Merge Windows (mergeWindows). 2. Trigger Handlers. a. Element (onElement). b. Merge (onMerge). 3. Trigger Timers. a. Processing Time (onProcessingTime). b. Event Time (onEventTime). 4. Evictor. a. Before (evictBefore). b. Evaluate Window (WindowFunction#apply).

@zimmermatt

Event processing flow.

slide-58
SLIDE 58

1. Window assigner. a. Assign Event to Window(s) (assignWindows). b. Merge Windows (mergeWindows). 2. Trigger Handlers. a. Element (onElement). b. Merge (onMerge). 3. Trigger Timers. a. Processing Time (onProcessingTime). b. Event Time (onEventTime). 4. Evictor. a. Before (evictBefore). b. Evaluate Window (WindowFunction#apply). c. After (evictAfter).

@zimmermatt

Event processing flow.

slide-59
SLIDE 59
  • Motivating Use Cases.
  • Window Requirements.
  • The Solution (Conceptual).
  • Event Processing Flow.
  • Apache Flink Window API Walk-Through.
  • The Solution in (Detail).
  • Pitfalls to Watch Out For.
  • Alternative Implementations.
  • Questions.

@zimmermatt

slide-60
SLIDE 60

@zimmermatt

Window API: Window.

package org.apache.flink.streaming.api.windowing.windows; public abstract class Window {

public abstract long maxTimestamp();

}

* If you implement Window, you’ll need to provide a TypeSerializer implementation for it.

slide-61
SLIDE 61

package org.apache.flink.streaming.api.windowing.assigners; public abstract class WindowAssigner<T, W extends Window> implements Serializable {

public abstract Collection<W> assignWindows(T element, long timestamp, WindowAssignerContext context);

public abstract Trigger<T, W> getDefaultTrigger(StreamExecutionEnvironment env); public abstract TypeSerializer<W> getWindowSerializer(ExecutionConfig executionConfig); public abstract boolean isEventTime(); public abstract static class WindowAssignerContext { public abstract long getCurrentProcessingTime(); } }

@zimmermatt

Window API: WindowAssigner.

slide-62
SLIDE 62

@zimmermatt

Window API: MergingWindowAssigner.

package org.apache.flink.streaming.api.windowing.assigners; public abstract class MergingWindowAssigner<T, W extends Window> extends WindowAssigner<T, W> {

public abstract void mergeWindows(Collection<W> windows, MergeCallback<W> callback);

public interface MergeCallback<W> { void merge(Collection<W> toBeMerged, W mergeResult); } }

slide-63
SLIDE 63

@zimmermatt

Window API: Trigger.

package org.apache.flink.streaming.api.windowing.triggers; public abstract class Trigger<T, W extends Window> implements Serializable { ...

public abstract TriggerResult onElement(T element, long timestamp, W window, TriggerContext ctx) throws Exception; public boolean canMerge() { return false; } public void onMerge(W window, OnMergeContext ctx) throws Exception { throws by default }

... }

slide-64
SLIDE 64

@zimmermatt

Window API: Trigger.

package org.apache.flink.streaming.api.windowing.triggers; public abstract class Trigger<T, W extends Window> implements Serializable { ...

public abstract TriggerResult onProcessingTime(long time, W window, TriggerContext ctx) throws Exception; public abstract TriggerResult onEventTime(long time, W window, TriggerContext ctx) throws Exception;

... }

slide-65
SLIDE 65

@zimmermatt

Window API: Trigger.

package org.apache.flink.streaming.api.windowing.triggers; public abstract class Trigger<T, W extends Window> implements Serializable { ... public abstract void clear(W window, TriggerContext ctx) throws Exception; public interface TriggerContext { ... } public interface OnMergeContext extends TriggerContext { ... } ... }

slide-66
SLIDE 66

@zimmermatt

Window API: Trigger.

package org.apache.flink.streaming.api.windowing.triggers; public abstract class Trigger<T, W extends Window> implements Serializable { ... public interface TriggerContext { long getCurrentProcessingTime(); MetricGroup getMetricGroup(); long getCurrentWatermark(); void registerProcessingTimeTimer(long time); void registerEventTimeTimer(long time); void deleteProcessingTimeTimer(long time); void deleteEventTimeTimer(long time); <S extends State> S getPartitionedState(StateDescriptor<S, ?> stateDescriptor); } public interface OnMergeContext extends TriggerContext { <S extends MergingState<?, ?>> void mergePartitionedState(StateDescriptor<S, ?> stateDescriptor); } }

slide-67
SLIDE 67

@zimmermatt

package org.apache.flink.streaming.api.windowing.evictors; public interface Evictor<T, W extends Window> extends Serializable {

void evictBefore(Iterable<TimestampedValue<T>> elements, int size, W window, EvictorContext evictorContext); void evictAfter(Iterable<TimestampedValue<T>> elements, int size, W window, EvictorContext evictorContext);

... }

Window API: Evictor.

slide-68
SLIDE 68

@zimmermatt

package org.apache.flink.streaming.api.windowing.evictors; public interface Evictor<T, W extends Window> extends Serializable { ... interface EvictorContext { long getCurrentProcessingTime(); MetricGroup getMetricGroup(); long getCurrentWatermark(); } }

Window API: Evictor.

slide-69
SLIDE 69

The solution

in detail.

@zimmermatt

slide-70
SLIDE 70

@zimmermatt

Custom Window: WindowAssigner.

public class CustomWindowAssigner<E extends CustomEvent> extends MergingWindowAssigner<E, CustomWindow<E>> { ... @Override public Collection<CustomWindow<E>> assignWindows(E element, long timestamp, WindowAssignerContext context) {

return Collections.singletonList(new CustomWindow<>(element, timeoutDuration));

} ... }

slide-71
SLIDE 71

@zimmermatt

Custom Window: Window.

public class CustomWindow<E extends CustomEvent> extends Window { ... @Override public long maxTimestamp() {

return maxTimestamp;

} ... }

slide-72
SLIDE 72

@zimmermatt

Custom Window: Window.

public class CustomWindow<E extends CustomEvent> extends Window { ...

@Override public boolean equals(Object o) { // important: equals implementation must compare using “value” semantics } @Override public int hashCode() { // important: same for hashCode implementation }

... }

slide-73
SLIDE 73

@zimmermatt

Custom Window: Window.

public class CustomWindow<E extends CustomEvent> extends Window { ...

public static class Serializer<T extends CustomEvent> extends TypeSerializer<CustomWindow<T>> { ... }

... }

slide-74
SLIDE 74

@zimmermatt

Custom Window: Window.

public class CustomWindow<E extends CustomEvent> extends Window { ...

public static class Serializer<T extends CustomEvent> extends TypeSerializer<CustomWindow<T>> {

@Override public boolean isImmutableType() { return true; }

... }

... }

slide-75
SLIDE 75

public class CustomWindow<E extends CustomEvent> extends Window { ... public static class Serializer<T extends CustomEvent> extends TypeSerializer<CustomWindow<T>> { ...

@Override public TypeSerializer<CustomWindow<T>> duplicate() { return this; } @Override public CustomWindow<T> createInstance() { return null; } @Override public CustomWindow<T> copy(CustomWindow<T> from) { return from; } @Override public CustomWindow<T> copy(CustomWindow<T> from, CustomWindow<T> reuse) { return from; } @Override public int getLength() { return -1; }

} ... }

@zimmermatt

Custom Window: Window.

slide-76
SLIDE 76

@zimmermatt

public class CustomWindow<E extends CustomEvent> extends Window { ...

public static class Serializer<T extends CustomEvent> extends TypeSerializer<CustomWindow<T>> { ...

public void serialize(CustomWindow<T> record, DataOutputView target) throws IOException { serializeStartEvent(record, target); target.writeLong(record.getDuration().toMillis()); target.writeBoolean(record.evaluate()); final boolean hasEndEventData = record.getEndEventData() != null; target.writeBoolean(hasEndEventData); if (hasEndEventData) serializeEndEvent(record, target); }

}

... }

Custom Window: Window.

slide-77
SLIDE 77

@zimmermatt

public class CustomWindow<E extends CustomEvent> extends Window { ... public static class Serializer<T extends CustomEvent> extends TypeSerializer<CustomWindow<T>> { ...

@Override public CustomWindow<T> deserialize(DataInputView source) throws IOException { final T startEvent = deserializeStartEvent(source); final Duration duration = Duration.ofMillis(source.readLong()); final boolean evaluate = source.readBoolean(); final boolean hasEndEventData = source.readBoolean(); final T endEvent = hasEndEventData ? deserializeEndEvent(source) : null; return new CustomWindow<>(startEvent, duration, endEvent, evaluate); }

} ... }

Custom Window: Window.

slide-78
SLIDE 78

@zimmermatt

public class CustomWindow<E extends CustomEvent> extends Window { ... public static class Serializer<T extends CustomEvent> extends TypeSerializer<CustomWindow<T>> { ...

@Override public CustomWindow<T> deserialize(CustomWindow<T> reuse, DataInputView source) throws IOException { return reuse != null ? reuse : deserialize(source); }

} ... }

Custom Window: Window.

slide-79
SLIDE 79

@zimmermatt

public class CustomWindow<E extends CustomEvent> extends Window { ... public static class Serializer<T extends CustomEvent> extends TypeSerializer<CustomWindow<T>> { ...

@Override public void copy(DataInputView source, DataOutputView target) throws IOException { // slightly less efficient, but more maintainable CustomWindow<T> deserializedWindow = deserialize(source); serialize(deserializedWindow, target); }

} ... }

Custom Window: Window.

slide-80
SLIDE 80

@zimmermatt

public class CustomWindow<E extends CustomEvent> extends Window { ... public static class Serializer<T extends CustomEvent> extends TypeSerializer<CustomWindow<T>> { ...

@Override public boolean equals(Object obj) { return obj instanceof Serializer; } @Override public boolean canEqual(Object obj) { return obj instanceof Serializer; } @Override public int hashCode() { return 0; }

} ... }

Custom Window: Window.

slide-81
SLIDE 81

@zimmermatt

public class CustomWindow<E extends CustomEvent> extends Window { ... public static class Serializer<T extends CustomEvent> extends TypeSerializer<CustomWindow<T>> { ...

@Override public TypeSerializerConfigSnapshot snapshotConfiguration() { ... } @Override public CompatibilityResult<CustomWindow<T>> ensureCompatibility( TypeSerializerConfigSnapshot configSnapshot) { return CompatibilityResult.requiresMigration(); } private static class CustomWindowSerializerConfigSnapshot extends TypeSerializerConfigSnapshot { ... }

} ... }

Custom Window: Window.

slide-82
SLIDE 82

@zimmermatt

Custom Window: Window.

public class CustomWindow<E extends CustomEvent> extends Window { … public CustomWindow(@Nonnull D primaryEventData, @Nonnull Duration timeoutDuration, D endEventData, boolean evaluate) { ...

this.endTimestamp = endEventData != null ? endEventData.getTimestamp() : maxTimestamp;

... } ...

public boolean evaluate() { return evaluate; } public Instant startTimestamp() { return primaryEventData.getTimestamp(); } public Instant endTimestamp() { return endTimestamp; }

} ... }

slide-83
SLIDE 83

@zimmermatt

Custom Window: WindowAssigner.

public class CustomWindowAssigner<E extends CustomEvent> extends MergingWindowAssigner<E, CustomWindow<E>> { ... @Override public void mergeWindows(Collection<CustomWindow<E>> mergeCandidates, MergeCallback<CustomWindow<E>> mergeCallback) {

final CustomWindow<E> sessionWindow = calculateSessionWindow(mergeCandidates); final Collection<CustomWindow<E>> inWindow = filterWithinWindow(mergeCandidates); // MergeCallback#merge implementation expects 2 or more. if (inWindow.size() > 1) { mergeCallback.merge(inWindow, sessionWindow); }

} ... }

slide-84
SLIDE 84

@zimmermatt

Custom Window: WindowAssigner.

public class CustomWindowAssigner<E extends CustomEvent> extends MergingWindowAssigner<E, CustomWindow<E>> { ... private CustomWindow<E> calculateSessionWindow(Collection<CustomWindow<E>> mergeCandidates) {

CustomWindow<E> startEventWindow = findStartEventWindow(mergeCandidates); if (startEventWindow != null) { // valid window … } else { // exploratory window ... }

} ... }

slide-85
SLIDE 85

@zimmermatt

Custom Window: WindowAssigner.

if (startEventWindow != null) { // valid window CustomWindow<E> endEvent = findEndEventWindow(mergeCandidates); // can return null return new CustomWindow<>(startEventWindow.getEvent, timeoutDuration, endEvent, true); // fire (send this one to the WindowFunction)

} else { // exploratory window ... }

slide-86
SLIDE 86

@zimmermatt

Custom Window: WindowAssigner.

if (startEventWindow != null) { // valid window ...

} else { // exploratory window CustomWindow<E> window = findClosestToMidpointByStartTime(mergeCandidates); return new CustomWindow(window.getEvent, exploratoryDuration, false) // just purge without firing }

slide-87
SLIDE 87

@zimmermatt

Time User A

t4 t3 t1 t2 t5

Watermark

slide-88
SLIDE 88

@zimmermatt

Time User A

t4 t3 t1 t2 t5

Watermark

slide-89
SLIDE 89

@zimmermatt

Time User A

t4 t3 t1 t2 t5

Watermark

slide-90
SLIDE 90

public class CustomWindowTrigger<E extends CustomEvent> extends Trigger<E, CustomWindow<E>> { ...

@Override public boolean canMerge() { return true; } @Override public void onMerge(CustomWindow<E> window, OnMergeContext onMergeContext) throws Exception {

  • nMergeContext.registerEventTimeTimer(window.endTimestamp().toEpochMilli());

}

... }

@zimmermatt

Custom Window: Trigger.

slide-91
SLIDE 91

@zimmermatt

Custom Window: Trigger.

public class CustomWindowTrigger<E extends CustomEvent> extends Trigger<E, CustomWindow<E>> { ...

@Override public TriggerResult onElement(E element, long timestamp, CustomWindow<E> window, TriggerContext triggerContext) throws Exception {

final TriggerResult triggerResult; final ValueState<Boolean> windowClosedState = triggerContext.getPartitionedState(windowClosedDescriptor); final long endTimestamp = window.endTimestamp().toEpochMilli();

if (triggerContext.getCurrentWatermark() >= endTimestamp) { triggerResult = windowClosedState.value() ? TriggerResult.CONTINUE : triggerWindow(triggerContext, windowClosedState, window);

} else { ... } return triggerResult;

} ... }

slide-92
SLIDE 92

@zimmermatt

Custom Window: Trigger.

public class CustomWindowTrigger<E extends CustomEvent> extends Trigger<E, CustomWindow<E>> { ...

private TriggerResult triggerWindow(TriggerContext triggerContext, ValueState<Boolean> windowClosedState, CustomWindow<E> window) throws IOException {

windowClosedState.update(Boolean.TRUE); removeEarlyFiringTimer(triggerContext); return window.evaluate() ? TriggerResult.FIRE_AND_PURGE : TriggerResult.PURGE;

} private void removeEarlyFiringTimer(TriggerContext triggerContext) throws IOException { final ValueState<Long> earlyFiringState = triggerContext.getPartitionedState(earlyFiringDescriptor);

if (earlyFiringState.value() > 0) { triggerContext.deleteProcessingTimeTimer(earlyFiringState.value()); // set to -1L to differentiate from the default value earlyFiringState.update(-1L); }

}

... }

slide-93
SLIDE 93

@zimmermatt

Custom Window: Trigger.

public class CustomWindowTrigger<E extends CustomEvent> extends Trigger<E, CustomWindow<E>> { ...

@Override public TriggerResult onElement(E element, long timestamp, CustomWindow<E> window, TriggerContext triggerContext) throws Exception { final TriggerResult triggerResult; final long endTimestamp = window.endTimestamp().toEpochMilli(); final ValueState<Boolean> windowClosedState = triggerContext.getPartitionedState(windowClosedDescriptor); if ...

} else { windowClosedState.update(Boolean.FALSE); triggerResult = TriggerResult.CONTINUE; triggerContext.registerEventTimeTimer(endTimestamp); registerEarlyFiringTimerIfNecessary(window, triggerContext); }

return triggerResult; }

... }

slide-94
SLIDE 94

@zimmermatt

Custom Window: Trigger.

public class CustomWindowTrigger<E extends CustomEvent> extends Trigger<E, CustomWindow<E>> { ...

private void registerEarlyFiringTimerIfNecessary(CustomWindow<E> window, TriggerContext triggerContext) throws IOException {

if (!window.evaluate() || earlyFiringInterval.toMillis() < 1) return;

final ValueState<Long> earlyFiringState = triggerContext.getPartitionedState(earlyFiringDescriptor); if (earlyFiringState.value() == Long.MIN_VALUE) { final Long newEarlyFiringTimestamp = System.currentTimeMillis() + earlyFiringInterval.toMillis(); if (newEarlyFiringTimestamp < window.endTimestamp().toEpochMilli()) { triggerContext.registerProcessingTimeTimer(newEarlyFiringTimestamp); earlyFiringState.update(newEarlyFiringTimestamp); } }

}

... }

slide-95
SLIDE 95

@zimmermatt

private void registerEarlyFiringTimerIfNecessary(CustomWindow<E> window, TriggerContext triggerContext) throws IOException { if (!window.evaluate() || earlyFiringInterval.toMillis() < 1) return;

final ValueState<Long> earlyFiringState = triggerContext.getPartitionedState(earlyFiringDescriptor); if (earlyFiringState.value() == Long.MIN_VALUE) { final Long newEarlyFiringTimestamp = System.currentTimeMillis() + earlyFiringInterval.toMillis(); if (newEarlyFiringTimestamp < window.endTimestamp().toEpochMilli()) { triggerContext.registerProcessingTimeTimer(newEarlyFiringTimestamp); earlyFiringState.update(newEarlyFiringTimestamp); }

}

Custom Window: Trigger.

slide-96
SLIDE 96

@zimmermatt

Custom Window: Trigger.

public class CustomWindowTrigger<E extends CustomEvent> extends Trigger<E, CustomWindow<E>> { ...

@Override public TriggerResult onEventTime(long time, CustomWindow<E> window, TriggerContext triggerContext) throws Exception {

if (time != window.endTimestamp().toEpochMilli()) { return TriggerResult.CONTINUE; }

final ValueState<Boolean> windowClosedState = triggerContext.getPartitionedState(windowClosedDescriptor); if (windowClosedState.value()) { return TriggerResult.CONTINUE; } return triggerWindow(triggerContext, windowClosedState, window); }

... }

slide-97
SLIDE 97

@zimmermatt

Custom Window: Trigger.

public class CustomWindowTrigger<E extends CustomEvent> extends Trigger<E, CustomWindow<E>> { ...

@Override public TriggerResult onEventTime(long time, CustomWindow<E> window, TriggerContext triggerContext) throws Exception { if (time != window.endTimestamp().toEpochMilli()) { return TriggerResult.CONTINUE; }

final ValueState<Boolean> windowClosedState = triggerContext.getPartitionedState(windowClosedDescriptor); if (windowClosedState.value()) { return TriggerResult.CONTINUE; } return triggerWindow(triggerContext, windowClosedState, window);

}

... }

slide-98
SLIDE 98

@zimmermatt

Custom Window: Trigger.

public class CustomWindowTrigger<E extends CustomEvent> extends Trigger<E, CustomWindow<E>> { ...

@Override public TriggerResult onProcessingTime(long time, CustomWindow<E> window, TriggerContext triggerContext) throws Exception {

TriggerResult triggerResult = TriggerResult.CONTINUE; if (window.evaluate()) { ... } return triggerResult; }

... }

slide-99
SLIDE 99

@zimmermatt

Custom Window: Trigger.

if (window.evaluate()) { // Update early firing final ValueState<Long> earlyFiringState = triggerContext.getPartitionedState(earlyFiringDescriptor); final Long newEarlyFiringTimestamp = earlyFiringState.value() + earlyFiringInterval.toMillis(); if (newEarlyFiringTimestamp < window.endTimestamp().toEpochMilli()) { triggerContext.registerProcessingTimeTimer(newEarlyFiringTimestamp); earlyFiringState.update(newEarlyFiringTimestamp); } triggerResult = TriggerResult.FIRE; } return triggerResult;

slide-100
SLIDE 100

@zimmermatt

  • Motivating Use Cases.
  • Window Requirements.
  • The Solution (Conceptual).
  • Event Processing Flow.
  • Apache Flink Window API Walk-Through.
  • The Solution (Detail).
  • Pitfalls to Watch Out For.
  • Alternative Implementations.
  • Questions.
slide-101
SLIDE 101

Pitfall:

@zimmermatt

Window equals / hashCode.

slide-102
SLIDE 102

Pitfall:

@zimmermatt

Metrics and Logs.

slide-103
SLIDE 103

Pitfall:

@zimmermatt

Event Design.

slide-104
SLIDE 104

Pitfall:

@zimmermatt

Large State.

slide-105
SLIDE 105

@zimmermatt

  • Motivating Use Cases.
  • Window Requirements.
  • The Solution (Conceptual).
  • Event Processing Flow.
  • Apache Flink Window API Walk-Through.
  • The Solution (Detail).
  • Pitfalls to Watch Out For.
  • Alternative Implementations.
  • Questions.
slide-106
SLIDE 106

Alternative:

@zimmermatt

ProcessFunction

slide-107
SLIDE 107

Alternative:

@zimmermatt

CEP Library

slide-108
SLIDE 108

@zimmermatt

  • Motivating Use Cases.
  • Window Requirements.
  • The Solution (Conceptual).
  • Event Processing Flow.
  • Apache Flink Window API Walk-Through.
  • The Solution (Detail).
  • Pitfalls to Watch Out For.
  • Alternative Implementations.
  • Questions.
slide-109
SLIDE 109

Thank You!

@zimmermatt