tencent cloud

Feedback

Practices on Loading JSON Data to Hive

Last updated: 2024-10-30 11:41:12

    1. Connect to Hive

    Log in to a master node of the EMR cluster, switch to the "hadoop" user, go to the Hive directory, and connect to Hive by running the following command:
    [root@10 ~]# su hadoop
    [hadoop@10 root]$ cd /usr/local/service/hive

    2. Prepare data

    Create a data file in JSON format. Compile the following code and save:
    vim test.data
    {"name":"Mary","age":12,"course":[{"name":"math","location":"b208"},{"name":"english","location":"b702"}],"grade":[99,98,95]}
    {"name":"Bob","age":20,"course":[{"name":"music","location":"b108"},{"name":"history","location":"b711"}],"grade":[91,92,93]}
    Store the data file in HDFS:
    hadoop fs -put ./test.data /

    3. Create a table

    Connect to Hive:
    [hadoop@10 hive]$ hive
    Create a table based on the mapping:
    hive> CREATE TABLE test (name string, age int, course array<map<string,string>>, grade array<int>) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' STORED AS TEXTFILE;

    4. Import data

    hive>LOAD DATA INPATH '/test.data' into table test;

    5. Check whether data import is successful

    Query all data:
    hive> select * from test;
    OK
    Mary 12 [{"name":"math","location":"b208"},{"name":"english","location":"b702"}] [99,98,95]
    Bob 20 [{"name":"music","location":"b108"},{"name":"history","location":"b711"}] [91,92,93]
    Time taken: 0.153 seconds, Fetched: 2 row(s)
    Query the first score of each record:
    hive> select grade[0] from test;
    OK
    99
    91
    Time taken: 0.374 seconds, Fetched: 2 row(s)
    Query the name and location of the first course of each record:
    hive> select course[0]['name'], course[0]['location'] from test;
    OK
    math b208
    music b108
    Time taken: 0.162 seconds, Fetched: 2 row(s)
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support