How to use groupByKey() on multiple RDDs?

I have multiple RDDs with one common field CustomerId.

CustomerId

For eg:

debitcardRdd has data as (CustomerId, debitField1, debitField2, ......)

debitcardRdd

(CustomerId, debitField1, debitField2, ......)

creditcardRdd has data as (CustomerId, creditField1, creditField2, ....)

creditcardRdd

(CustomerId, creditField1, creditField2, ....)

netbankingRdd has data as (CustomerId, nbankingField1, nbankingField2, ....)

netbankingRdd

(CustomerId, nbankingField1, nbankingField2, ....)

We perform different transformations on each individual rdd, however we need to perform a transformation on the data from all the 3 rdds by grouping CustomerId.

CustomerId

Example: (CustomerId,debitFiedl1,creditField2,bankingField1,....)

(CustomerId,debitFiedl1,creditField2,bankingField1,....)

Is there any way we can group the data from all RDDs based on same key.

Note: In Apache Beam it can be done by using coGroupByKey, just checking if there is such alternative available in spark.

coGroupByKey

RDD[K, V]

2 Answers
2

Just cogroup

cogroup

debitcardRdd.keyBy(_.CustomerId).cogroup(
creditcardRdd.keyBy(_.CustomerId),
netbankingRdd.keyBy(_.CustomerId)
)


In contrast to the below, the .keyBy is not imho actually required here and we note that cogroup - not well described can extend to n RDDs.

val rddREScogX = rdd1.cogroup(rdd2,rdd3,rddn, ...)


Points should go to the first answer.

Thanks for contributing an answer to Stack Overflow!

But avoid …

To learn more, see our tips on writing great answers.

Required, but never shown

Required, but never shown

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.