Dfyjkt

I want to find unique List[String] from an Array[List[String]].

For example,
suppose we have an Array of the following List[String]

[a, b, c]
[a, b]
[a, b]
[a, c]

The expected result would be

[a, b, c]
[a, b]
[a, c]

Yes. You can apply .distinct in Array(List(String))

def distinct: Array[List[String]]

Builds a new mutable indexed sequence from this mutable indexed
sequence without any duplicate elements.

Returns

A new mutable indexed sequence which contains the first occurrence of
every element of this mutable indexed sequence.

Try the below snippet

import org.apache.spark.sql.SparkSession

object StackTest 
 def main(args: Array[String]): Unit = 

 System.setProperty("hadoop.home.dir", "C:\hadoop")
 val spark = SparkSession
 .builder()
 .config("spark.master", "local[1]")
 .appName("StackOverFlow")
 .getOrCreate()

 spark.sparkContext.setLogLevel("WARN")

 val hc = spark.sqlContext
 import spark.implicits._

 //Define Array[List[String]]
 var myArrList = Array(List("a","b","c"),List("a","b"),List("a","b"),List("a","c"))
 println("ArrayList: "+ myArrList.deep)

 var distinctMyArrList = myArrList.distinct
 println("Distinct ArrayList: "+ distinctMyArrList.deep)

 

OUTPUT

ArrayList: Array(List(a, b, c), List(a, b), List(a, b), List(a, c))
Distinct ArrayList: Array(List(a, b, c), List(a, b), List(a, c))