spark 中hint使用总结

在spark sql 中用户可以使用Join hint来建议Spark使用哪一种Join。在Spark 3.0以前,只支持BROADCAST这种Join hint。从Spark 3.0开始增加了MERGE、SHUFFLE_HASH和SHUFFLE_REPLICATE_NL这三种Join Hint。优先级为BROADCAST > MERGE > SHUFFLE_HASH > SHUFFLE_REPLICATE_NL。如果Join的两侧都添加了BROADCAST或者SHUFFLE_HASH,则Spark会根据joinType和两侧的大小来选择build哪一侧。

-- Join Hints for broadcast join
SELECT /*+ BROADCAST(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;
SELECT /*+ BROADCASTJOIN (t1) */ * FROM t1 left JOIN t2 ON t1.key = t2.key;
SELECT /*+ MAPJOIN(t2) */ * FROM t1 right JOIN t2 ON t1.key = t2.key;

-- Join Hints for shuffle sort merge join
SELECT /*+ SHUFFLE_MERGE(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;
SELECT /*+ MERGEJOIN(t2) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;
SELECT /*+ MERGE(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;

-- Join Hints for shuffle hash join
SELECT /*+ SHUFFLE_HASH(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;

-- Join Hints for shuffle-and-replicate nested loop join
SELECT /*+ SHUFFLE_REPLICATE_NL(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;

-- When different join strategy hints are specified on both sides of a join, Spark
-- prioritizes the BROADCAST hint over the MERGE hint over the SHUFFLE_HASH hint
-- over the SHUFFLE_REPLICATE_NL hint.
-- Spark will issue Warning in the following example
-- org.apache.spark.sql.catalyst.analysis.HintErrorLogger: Hint (strategy=merge)
-- is overridden by another hint and will not take effect.
SELECT /*+ BROADCAST(t1), MERGE(t1, t2) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;

spark hint 中使用关系:https://blog.51cto.com/u_15435003/5296344

相关推荐

  1. spark hint使用总结

    2024-07-12 16:16:03       22 阅读
  2. Spark广播的使用

    2024-07-12 16:16:03       27 阅读
  3. spark 报错总结

    2024-07-12 16:16:03       42 阅读
  4. WPFMVVM使用总结

    2024-07-12 16:16:03       53 阅读
  5. 使用spark mllib训练文文本分类器的

    2024-07-12 16:16:03       56 阅读

最近更新

  1. docker php8.1+nginx base 镜像 dockerfile 配置

    2024-07-12 16:16:03       66 阅读
  2. Could not load dynamic library ‘cudart64_100.dll‘

    2024-07-12 16:16:03       70 阅读
  3. 在Django里面运行非项目文件

    2024-07-12 16:16:03       57 阅读
  4. Python语言-面向对象

    2024-07-12 16:16:03       68 阅读

热门阅读

  1. 安卓文件上传照片单张及多张照片上传实现

    2024-07-12 16:16:03       17 阅读
  2. 编译Linux内核, 制作迷你系统并在虚拟机里运行

    2024-07-12 16:16:03       21 阅读
  3. 力扣1209.删除字符串中的所有相邻重复项 II

    2024-07-12 16:16:03       20 阅读
  4. Python使用总结之jieba形容词提取详解

    2024-07-12 16:16:03       21 阅读
  5. postman接口测试工具详解

    2024-07-12 16:16:03       21 阅读
  6. 迅为RK3568手册上新 | RK3568开发板NPU例程测试

    2024-07-12 16:16:03       21 阅读
  7. Yolo的离线运行

    2024-07-12 16:16:03       23 阅读
  8. 2024.07.04校招 实习 内推 面经

    2024-07-12 16:16:03       20 阅读
  9. 从零开始学习嵌入式----C语言指针函数

    2024-07-12 16:16:03       18 阅读