HIVE部署

2019-12-24

前言

部署hive需要提前部署好以下环境：

mysql （安装在10.57.22.113）
HDFS (hadoop-2.7.7)

一、搭建模式

Hive中搭建分如下3中方式，三种方式归根到底就是元数据的存储位置不一样

1
2
3

a)内嵌Derby方式 ：运行hive会在当前目录生成一个derby文件和一个metastore_db目录。这种存储方式的弊端是在同一个目录下同时只能有一个hive客户端能使用数据库。
b)mysql Local方式 (hive.metastore.uris为空表示嵌入模式或本地模式)
c)多用户模式 (mysql Remote方式; hive.metastore.uris不为空为远程模式)

二、部署Hive

A）下载地址：https://archive.apache.org/dist/hive/hive-2.3.4/apache-hive-2.3.4-bin.tar.gz

B）解压包：tar -xvf apache-hive-2.3.4-bin.tar.gz

C）修改文件夹名称：mv apache-hive-2.3.4-bin hive-2.3.4

D）配置环境变量, 修改.bashrc文件

vi ~/.bashrc

1 2	export HIVE_HOME=/data01/bigdata/hive-2.3.4 export PATH=$PATH:$HIVE_HOME/bin

source ~/.bashrc

三、配置修改

A）修改hive-site.xml配置

首先复制一份配置文件cp hive-default.xml.template hive-site.xml，修改如下配置：

<!-- 指定HDFS中的hive仓库地址 --> 
 <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/data01/bigdata/hive-2.3.4/warehouse</value>
    <description>location of default database for the warehouse</description>
  </property>
  
  <property>
    <name>hive.exec.scratchdir</name>
    <value>/data01/bigdata/hive-2.3.4</value>
    <description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/&lt;username&gt; is created, with ${hive.scratch.dir.permission}.</description>
  </property>

<!-- 指定驱动类 -->
   <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://10.57.22.113/hive_remote?createDatabaseIfNotExist=true</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
    </description>

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
    <description>Username to use against metastore database</description>
  </property>

   <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>root</value>
    <description>password to use against metastore database</description>
  </property>

<!-- 该属性为空表示嵌入模式或本地模式，否则为远程模式 -->  
   <property>
    <name>hive.metastore.uris</name>
    <value>thrift://10.57.22.119:9083</value>
    <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
  </property>
  
  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/tmp/hive/local</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
  
  <property>
    <name>hive.downloaded.resources.dir</name>
    <value>/tmp/hive/resources</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
  </property>
 
<property>
    <name>hive.server2.webui.host</name>
    <value>tdcdh119</value>
    <description>The host address the HiveServer2 WebUI will listen on</description>
  </property>
  <property>
    <name>hive.server2.webui.port</name>
    <value>10002</value>
    <description>The port the HiveServer2 WebUI will listen on. This can beset to 0 or a negative integer to disable the web UI</description>
  </property>

B) 修改hive-env.sh变量

首先复制一份配置文件cp hive-env.sh.template hive-env.sh, 新增如下配置：

1
2
3

export  HADOOP_HOME=/data01/bigdata/hive-2.3.4
export  HIVE_CONF_DIR=/data01/bigdata/hive-2.3.4/conf
export  HIVE_AUX_JARS_PATH=/data01/bigdata/hive-2.3.4lib

四、其他配置

A）新建warehouse目录

mkdir /data01/bigdata/hive-2.3.4/warehouse

将warehouse变为hadoop仓库
hadoop fs -mkdir -p /data01/bigdata/hive-2.3.4/warehouse
hadoop fs -chmod 777 /data01/bigdata/hive-2.3.4/warehouse

B）上传mysql-connector-java-5.1.40.jar到指定目录$HADOOP_HOME/lib

1	scp ~/Downloads/mysql-connector-java-5.1.40.jar admin@10.57.22.119:/data01/bigdata/hive-2.3.4/lib

C) 创建存储元数据的mysql库

1 2	mysql -h 10.57.22.113 -u root -p create database hive_remote;

D) 初始化schema（/data01/bigdata/hive-2.3.4/bin/schematool）

1	schematool -dbType mysql -initSchema

进入mysql查看初始化schema后结果，如下：

mysql> use hive_remote;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+---------------------------+
| Tables_in_hive_remote     |
+---------------------------+
| aux_table                 |
| bucketing_cols            |
| cds                       |
| columns_v2                |
| compaction_queue          |
| completed_compactions     |
| completed_txn_components  |
| database_params           |
| db_privs                  |
| dbs                       |
| delegation_tokens         |
| func_ru                   |
| funcs                     |
| global_privs              |
| hive_locks                |
| idxs                      |
| index_params              |
| key_constraints           |
| master_keys               |
| next_compaction_queue_id  |
| next_lock_id              |
| next_txn_id               |
| notification_log          |
| notification_sequence     |
| nucleus_tables            |
| part_col_privs            |
| part_col_stats            |
| part_privs                |
| partition_events          |
| partition_key_vals        |
| partition_keys            |
| partition_params          |
| partitions                |
| role_map                  |
| roles                     |
| sd_params                 |
| sds                       |
| sequence_table            |
| serde_params              |
| serdes                    |
| skewed_col_names          |
| skewed_col_value_loc_map  |
| skewed_string_list        |
| skewed_string_list_values |
| skewed_values             |
| sort_cols                 |
| tab_col_stats             |
| table_params              |
| tbl_col_privs             |
| tbl_privs                 |
| tbls                      |
| txn_components            |
| txns                      |
| type_fields               |
| types                     |
| version                   |
| write_set                 |
+---------------------------+
57 rows in set (0.06 sec)

五、启动hive server

启动 hive server
nohup hive --service metastore &

启动hive web UI
nohup hive --service hiveserver2 &

进程查看,是否有RunJar进程（有即服务已启动）：

1
2
3

[admin@tdcdh119 warehouse]$ jps
17627 RunJar
19643 Jps

六、启动hive client

hive

验证hive

hive> create databases hive_test;
OK
Time taken: 1.184 seconds
hive> show databases;
OK
default
hive_test
Time taken: 1.131 seconds, Fetched: 2 row(s)

七、参考文档

https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-InstallingHivefromaStableRelease

https://www.cnblogs.com/LHWorldBlog/p/8289702.html

https://www.cnblogs.com/dxxblog/p/8193967.html

https://www.cnblogs.com/xuwujing/p/8045821.html

https://blog.csdn.net/zhangxiaohui4445/article/details/90675445