blog/BigData-Notes
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update Flink_Data_Transformation.md

Browse Source
This commit is contained in:
heibaiying 2019-10-26 20:25:30 +08:00 committed by GitHub
parent 96d6951d24
commit 948c0dad74
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 14 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
notes/Flink_Data_Transformation.md
View File

@ -65,7 +65,10 @@ env.fromElements(1, 2, 3, 4, 5).filter(x -> x > 3).print();
如下例子将数据按照 key 值分区后，滚动进行求和计算： 如下例子将数据按照 key 值分区后，滚动进行求和计算：
```java ```java
DataStream<Tuple2<String, Integer>> tuple2DataStream = env.fromElements(new Tuple2<>("a", 1), new Tuple2<>("a", 2), new Tuple2<>("b", 3), new Tuple2<>("b", 5)); DataStream<Tuple2<String, Integer>> tuple2DataStream = env.fromElements(new Tuple2<>("a", 1),
new Tuple2<>("a", 2),
new Tuple2<>("b", 3),
new Tuple2<>("b", 5));
KeyedStream<Tuple2<String, Integer>, Tuple> keyedStream = tuple2DataStream.keyBy(0); KeyedStream<Tuple2<String, Integer>, Tuple> keyedStream = tuple2DataStream.keyBy(0);
keyedStream.reduce((ReduceFunction<Tuple2<String, Integer>>) (value1, value2) -> keyedStream.reduce((ReduceFunction<Tuple2<String, Integer>>) (value1, value2) ->
new Tuple2<>(value1.f0, value1.f1 + value2.f1)).print(); new Tuple2<>(value1.f0, value1.f1 + value2.f1)).print();
@ -113,8 +116,10 @@ keyedStream.maxBy("key");
用于连接两个或者多个元素类型相同的 DataStream 。当然一个 DataStream 也可以与其本生进行连接，此时该 DataStream 中的每个元素都会被获取两次： 用于连接两个或者多个元素类型相同的 DataStream 。当然一个 DataStream 也可以与其本生进行连接，此时该 DataStream 中的每个元素都会被获取两次：
```shell ```shell
DataStreamSource<Tuple2<String, Integer>> streamSource01 = env.fromElements(new Tuple2<>("a", 1), new Tuple2<>("a", 2)); DataStreamSource<Tuple2<String, Integer>> streamSource01 = env.fromElements(new Tuple2<>("a", 1),
DataStreamSource<Tuple2<String, Integer>> streamSource02 = env.fromElements(new Tuple2<>("b", 1), new Tuple2<>("b", 2)); new Tuple2<>("a", 2));
DataStreamSource<Tuple2<String, Integer>> streamSource02 = env.fromElements(new Tuple2<>("b", 1),
new Tuple2<>("b", 2));
streamSource01.union(streamSource02); streamSource01.union(streamSource02);
streamSource01.union(streamSource01,streamSource02); streamSource01.union(streamSource01,streamSource02);
``` ```
@ -124,7 +129,8 @@ streamSource01.union(streamSource01,streamSource02);
Connect 操作用于连接两个或者多个类型不同的 DataStream ，其返回的类型是 ConnectedStreams ，此时被连接的多个 DataStreams 可以共享彼此之间的数据状态。但是需要注意的是由于不同 DataStream 之间的数据类型是不同的，如果想要进行后续的计算操作，还需要通过 CoMap 或 CoFlatMap 将 ConnectedStreams 转换回 DataStream： Connect 操作用于连接两个或者多个类型不同的 DataStream ，其返回的类型是 ConnectedStreams ，此时被连接的多个 DataStreams 可以共享彼此之间的数据状态。但是需要注意的是由于不同 DataStream 之间的数据类型是不同的，如果想要进行后续的计算操作，还需要通过 CoMap 或 CoFlatMap 将 ConnectedStreams 转换回 DataStream：
```java ```java
DataStreamSource<Tuple2<String, Integer>> streamSource01 = env.fromElements(new Tuple2<>("a", 3), new Tuple2<>("b", 5)); DataStreamSource<Tuple2<String, Integer>> streamSource01 = env.fromElements(new Tuple2<>("a", 3),
new Tuple2<>("b", 5));
DataStreamSource<Integer> streamSource02 = env.fromElements(2, 3, 9); DataStreamSource<Integer> streamSource02 = env.fromElements(2, 3, 9);
// 使用connect进行连接 // 使用connect进行连接
ConnectedStreams<Tuple2<String, Integer>, Integer> connect = streamSource01.connect(streamSource02); ConnectedStreams<Tuple2<String, Integer>, Integer> connect = streamSource01.connect(streamSource02);
@ -170,7 +176,9 @@ split.select("even").print();
project 主要用于获取 tuples 中的指定字段集，示例如下： project 主要用于获取 tuples 中的指定字段集，示例如下：
```java ```java
DataStreamSource<Tuple3<String, Integer, String>> tuple3DataStreamSource = env.fromElements(new Tuple3<>("li", 22, "2018-09-23"),new Tuple3<>("ming", 33, "2020-09-23")); DataStreamSource<Tuple3<String, Integer, String>> tuple3DataStreamSource = env.fromElements(
new Tuple3<>("li", 22, "2018-09-23"),
new Tuple3<>("ming", 33, "2020-09-23"));
tuple3DataStreamSource.project(0,2).print(); tuple3DataStreamSource.project(0,2).print();
// 输出 // 输出
@ -284,4 +292,4 @@ someStream.filter(...).slotSharingGroup("slotSharingGroupName");
## 参考资料 ## 参考资料
Flink Operators： https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/ Flink Operators： https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/