Quantcast
Channel: Active questions tagged consumer - Stack Overflow
Viewing all articles
Browse latest Browse all 87

how to load JSON into hive table?

$
0
0

i have a python code with consumes messages from Kafka topic. I want to store these messages into hive table.

import sslfrom kafka import KafkaConsumer, TopicPartitionfrom pyhive import hivessl.match_hostname = lambda cert, hostname: Truemytopic='xyz'mygroupid='ABC'mytopicpartitions = 10print("start kafka connection")consumer = KafkaConsumer(bootstrap_servers='xxx:xx', group_id=mygroupid, security_protocol='SSL',s ssl_check_hostname=True, ssl_cafile='xx.pem', ssl_certfile='certificate.pem', ssl_keyfile='key.pem', auto_offset_reset='latest', consumer_timeout_ms=60000)print("end kafka connection")# Set topic and partitiontopicPartition=[TopicPartition(mytopic, p) for p in range(0,mytopicpartitions)]consumer.assign(topicPartition)consumer.seek_to_beginning()conn = hive.Connection(host="00000", port=1111, username="user",password="pass",auth="CUSTOM")cursor = conn.cursor()query=("insert into table sample values('{}')".format(consumer))cursor.execute(query)

error: its loading the object <kafka.consumer.group.KafkaConsumer object at 0x7f88fe848d30>not the actual data.


Viewing all articles
Browse latest Browse all 87

Trending Articles