Dfyjkt

I have issue with the performance when reading data from HBase in spark streaming. It is taking more than 5 mins just to read data from HBase for 3 records. Below is the logic that I used in mapPartitions.

val messages = KafkaUtils.createDirectStream[String, Array[Byte], StringDecoder, DefaultDecoder](ssc, kafkaParams, topicSet)
messages.mapPartitions(iter => 

 val context = TaskContext.get
 logger.info((s"Process for partition: $context.partitionId "))

 val hbaseConf = HBaseConfiguration.create()
 //hbaseConf.addResource(new File("/etc/hbase/conf/hbase-site.xml").toURI.toURL) 
 //val connection: Connection = hbaseConnection.getOrCreateConnection(hbaseConf)
 val connection = ConnectionFactory.createConnection(hbaseConf)
 val hbaseTable = connection.getTable(TableName.valueOf("prod:CustomerData"))

 .......
 )

I have used BulkGet. It is taking around 5 seconds to process 90K messages(may be because the API is using HBaseContext and we dont have create any HBaseConnection). But I cannot use this as the output of BulkGet is RDD, and I have to do leftouterjoin to join the RDD of BulkGet with the actual RDD from Kafka. I assume this is not correct approach as it involves the below. Moreover I have to process all the 90K messages in 1 second.

1. Fetch distinct Cusotmer Id from the RDD read from Kafka before passing it to BulkGet




2. Also, it involves shuffling as I have to do leftOuterJoin the main RDD (from Kafka) with the BulkGet RDD (I only see the option of join as the BulkGet output is an RDD)

Can anyone please help me what is the issue with performance when I try to create HBaseConnection in mapPartitions. I have also tried setting driver-class-path.

Thanks