Distinct window functions are not supported: count(distinct _w0# ) windowspecdefinition_编程学问网

Distinct window functions are not supported: count(distinct _w0# ) windowspecdefinition

spark | 2021-05-14 17:24:28

spark 开窗使用 countDistinct异常

Distinct window functions are not supported: count(distinct _w0# ) windowspecdefinition

groupby 可以直接使用countDistinct，但是 over(Window.partionBy) 不可以

方案一、可以使用approx_count_distinct，但是它是概率统计，非精确统计，其是使用HyperLogLog实现的

import org.apache.spark.sql.functions._

val windowSpec = Window.partitionBy($"site_id",$"pxSessionId").orderBy($"timeMs").rowsBetween(Long.MinValue,Long.MaxValue)

res.withColumn("pageViewNumber",approx_count_distinct(when($"event_name" === "pageview",$"pxSessionId")).over(windowSpec))

方案二（推荐）、collect_set与size结合使用，示例如下

import org.apache.spark.sql.functions._

val windowSpec = Window.partitionBy($"site_id",$"pxSessionId").orderBy($"timeMs").rowsBetween(Long.MinValue,Long.MaxValue)

res.withColumn("pageViewNumber",size(collect_set(when($"event_name" === "pageview",$"pxSessionId")).over(windowSpec)))

登录后即可回复登录 | 注册

相关文章

mysql in和not in查询速度优化方法 css animation iteration count php eventhttp sets default callback to handle requests that are not caught by specific callbacks php mysqlnd qc 函数 sets the callback functions for a user defined procedural storage handler php ncurses 函数 remove panel from the stack and delete it but not the associated window php oci8 obsolete aliases and functions 别名 oci field size php oci8 obsolete aliases and functions 别名 oci close php oci8 obsolete aliases and functions 别名 oci statement type php pdf 函数 open image created with php s image functions not supported php socket 函数 sends a message to a socket whether it is connected or not php mongocollection creates an index on the specified field s if it does not already exist php mysql xdevapi baseresult fetch warning count from last operation php what makes a phar a phar and not a tar or a zip? zip based phars php pdo 驱动 4d functions pdo 4d php solrquery returns whether or not to include the lower bound in the result set php sphinxclient set attribute name for per group distinct values count calculations php swoole connection iterator count connections.php swoole event add new callback functions of a socket into the eventloop.Distinct window functions are not supported: count(distinct _w0# ) windowspecdefinition

关注编程学问公众号