Distinct window functions are not supported: count(distinct _w0# ) windowspecdefinition_编程学问网

Distinct window functions are not supported: count(distinct _w0# ) windowspecdefinition

spark | 2021-05-14 17:24:28

spark 开窗使用 countDistinct异常

Distinct window functions are not supported: count(distinct _w0# ) windowspecdefinition

groupby 可以直接使用countDistinct，但是 over(Window.partionBy) 不可以

方案一、可以使用approx_count_distinct，但是它是概率统计，非精确统计，其是使用HyperLogLog实现的

import org.apache.spark.sql.functions._

val windowSpec = Window.partitionBy($"site_id",$"pxSessionId").orderBy($"timeMs").rowsBetween(Long.MinValue,Long.MaxValue)

res.withColumn("pageViewNumber",approx_count_distinct(when($"event_name" === "pageview",$"pxSessionId")).over(windowSpec))

方案二（推荐）、collect_set与size结合使用，示例如下

import org.apache.spark.sql.functions._

val windowSpec = Window.partitionBy($"site_id",$"pxSessionId").orderBy($"timeMs").rowsBetween(Long.MinValue,Long.MaxValue)

res.withColumn("pageViewNumber",size(collect_set(when($"event_name" === "pageview",$"pxSessionId")).over(windowSpec)))

登录后即可回复登录 | 注册

相关文章

jintellitype异常could not load jintellitype.dll php enchant 函数 will return a list of values if any of those pre conditions are not met php informix 函数 count the rows already fetched from a query php ncurses 函数 remove panel from the stack and delete it but not the associated window php oci8 obsolete aliases and functions 别名 oci collection append php oci8 obsolete aliases and functions 别名 oci error php oci8 obsolete aliases and functions 别名 oci new connect php pdf 函数 open image created with php s image functions not supported php runkit 函数 determines if the current functions return value will be used php sqlite 函数 execute a query that does not prefetch and buffer all data php mongodb driver cursorid create a new cursorid not used php mysql xdevapi rowresult get warning count from last operation php pharfileinfo returns crc32 code or throws an exception if crc has not been verified php pdo 驱动 informix functions pdo informix php sphinxclient set attribute name for per group distinct values count calculations php splfileobject not at eof php more details functions and statements which will check tainted string php yaf session the count purpose Distinct window functions are not supported: count(distinct _w0# ) windowspecdefinition

关注编程学问公众号