前言
大家好，我是明哥!
本片博文是”大数据问题排查系列”之一，讲述某HiVE SQL 作业因为 HiVE 中的元数据与 HDFS中实际的数据不一致引起的一个问题的排查和修复。
以下是正文。
问题现象
客户端报错如下：
unable to MOVe souRce xxx to destination xxx
客户端报错
问题分析
客户端的报错信息，并没有完全展现问题背后的全貌。我们进入 HiveseRveR2 所在节点查看HiveseRveR2的日志，可以看到如下相关信息：
2021-09-01 11:47:46,795 INFO oRg.Apache.hadoop.Hive.ql.exec.Task: [HiveSeRveR2-background-Pool: ThRead-1105]: loading data to table hs_ods.ods_ses_acct_aSSuRe_scale paRtITion (paRt_date=20210118) fRoM hdfs://hs01:8020/User/hundsun/dap/Hive/hs_ods/ods_ses_acct_aSSuRe_scale/paRt_date=20210118/.Hive-staging_Hive_2021-09-01_11-47-31_216_694180642957006705-35/-ext-10000 2021-09-01 11:47:46,795 INFO Hive.MetasTore: [HiveSeRveR2-background-Pool: ThRead-1105]: HMS client filteRing is enabled. 2021-09-01 11:47:46,795 INFO Hive.MetasTore: [HiveSeRveR2-background-Pool: ThRead-1105]: trying to connect to MetasTore wITh URI thRift://hs01:9083 2021-09-01 11:47:46,795 INFO Hive.MetasTore: [HiveSeRveR2-background-Pool: ThRead-1105]: Opened a connection to MetasTore, cuRRent Connections: 54 2021-09-01 11:47:46,796 INFO Hive.MetasTore: [HiveSeRveR2-background-Pool: ThRead-1105]: Connected to MetasTore. 2021-09-01 11:47:46,928 INFO oRg.Apache.hadoop.Hive.ql.exec.MOVeTask: [HiveSeRveR2-background-Pool: ThRead-1105]: PaRtITion is: {paRt_date=20210118} 2021-09-01 11:47:46,945 INFO oRg.Apache.hadoop.Hive.coMMon.fileUtils: [HiveSeRveR2-background-Pool: ThRead-1105]: CReating diRecTory if IT doesn’t exist: hdfs://hs01:8020/User/hundsun/dap/Hive/hs_ods/ods_ses_acct_aSSuRe_scale/paRt_date=20210118 2021-09-01 11:47:46,947 Error Hive.ql.Metadata.Hive: [HiveSeRveR2-background-Pool: ThRead-1105]: FAIled to MOVe: {} 2021-09-01 11:47:46,947 Error oRg.Apache.hadoop.Hive.ql.DRiveR: [HiveSeRveR2-background-Pool: ThRead-1105]: Failed: Execution Error, RetuRn code 1 fRoM oRg.Apache.hadoop.Hive.ql.exec.MOVeTask. unable to MOVe souRce hdfs://hs01:8020/User/hundsun/dap/Hive/hs_ods/ods_ses_acct_aSSuRe_scale/paRt_date=20210118/.Hive-staging_Hive_2021-09-01_11-47-31_216_694180642957006705-35/-ext-10000 to destination hdfs://hs01:8020/User/hundsun/dap/Hive/hs_ods/ods_ses_acct_aSSuRe_scale/paRt_date=20210118 2021-09-01 11:47:46,948 INFO oRg.Apache.hadoop.Hive.ql.DRiveR: [HiveSeRveR2-background-Pool: ThRead-1105]: CoMpleted executing coMMand(queRyId=Hive_20210901114731_d7a78302-fb2a-4b45-9472-db6a9787f710); TiMe taken: 15.489 seconds 2021-09-01 11:47:46,957 Error oRg.Apache.Hive.seRvice.cli.operation.operation: [HiveSeRveR2-background-Pool: ThRead-1105]: Error Running Hive queRy: oRg.Apache.Hive.seRvice.cli.HiveSQLException: Error wHile ProceSSing stateMent: Failed: Execution ERRoR, RetuRn code 1 fRoM oRg.Apache.hadoop.Hive.ql.exec.MOVeTask. unable to MOVe souRce hdfs://hs01:8020/User/hundsun/dap/Hive/hs_ods/ods_ses_acct_aSSuRe_scale/paRt_date=20210118/.Hive-staging_Hive_2021-09-01_11-47-31_216_694180642957006705-35/-ext-10000 to destination hdfs://hs01:8020/User/hundsun/dap/Hive/hs_ods/ods_ses_acct_aSSuRe_scale/paRt_date=20210118 at oRg.Apache.Hive.seRvice.cli.operation.operation.toSQLException(operation.java:329) ~[Hive-seRvice-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.Hive.seRvice.cli.operation.SQLoperation.RunQueRy(SQLoperation.java:258) ~[Hive-seRvice-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.Hive.seRvice.cli.operation.SQLoperation.acceSS$600(SQLoperation.java:92) ~[Hive-seRvice-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.Hive.seRvice.cli.operation.SQLoperation$backgroundWoRk$1.Run(SQLoperation.java:345) [Hive-seRvice-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at java.security.AcceSSContRolleR.doPRivileged(Native method) ~[?:1.8.0_271] at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_271] at oRg.Apache.hadoop.secuRITy.UserGRoupinformation.doAs(useRGRoupinformation.java:1875) [hadoop-coMMon-3.0.0-cdh6.3.2.jaR:?] at oRg.Apache.Hive.seRvice.cli.operation.SQLoperation$backgroundWoRk.Run(SQLoperation.java:357) [Hive-seRvice-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at java.util.concuRRent.ExecuTors$RunnableAdapteR.call(ExecuTors.java:511) [?:1.8.0_271] at java.util.concuRRent.FutuReTask.Run(FutuReTask.java:266) [?:1.8.0_271] at java.util.concuRRent.ThReadPoolExecuTor.RunWoRkeR(ThReadPoolExecuTor.java:1149) [?:1.8.0_271] at java.util.concuRRent.ThReadPoolExecuTor$WoRkeR.Run(ThReadPoolExecuTor.java:624) [?:1.8.0_271] at java.lang.ThRead.Run(ThRead.java:748) [?:1.8.0_271] Caused by: oRg.Apache.hadoop.Hive.ql.Metadata.HiveException: unable to MOVe souRce hdfs://hs01:8020/User/hundsun/dap/Hive/hs_ods/ods_ses_acct_aSSuRe_scale/paRt_date=20210118/.Hive-staging_Hive_2021-09-01_11-47-31_216_694180642957006705-35/-ext-10000 to destination hdfs://hs01:8020/User/hundsun/dap/Hive/hs_ods/ods_ses_acct_aSSuRe_scale/paRt_date=20210118 at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.getHiveException(Hive.java:3449) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.getHiveException(Hive.java:3405) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.MOVefile(Hive.java:3400) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.ReplACEfiles(Hive.java:3697) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.loadPaRtITionInteRnal(Hive.java:1614) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.loadPaRtITion(Hive.java:1525) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.loadPaRtITion(Hive.java:1489) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.exec.MOVeTask.execute(MOVeTask.java:501) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.exec.Task.executeTask(Task.java:199) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.exec.TaskRunneR.RunSequential(TaskRunneR.java:97) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.DRiveR.launchTask(DRiveR.java:2200) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.DRiveR.execute(DRiveR.java:1843) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.DRiveR.RunInteRnal(DRiveR.java:1563) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.DRiveR.Run(DRiveR.java:1339) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.DRiveR.Run(DRiveR.java:1334) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.Hive.seRvice.cli.operation.SQLoperation.RunQueRy(SQLoperation.java:256) ~[Hive-seRvice-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] … 11 MoRe Caused by: oRg.Apache.hadoop.Hive.ql.Metadata.HiveException: unable to MOVe souRce hdfs://hs01:8020/User/hundsun/dap/Hive/hs_ods/ods_ses_acct_aSSuRe_scale/paRt_date=20210118/.Hive-staging_Hive_2021-09-01_11-47-31_216_694180642957006705-35/-ext-10000/000000_0 to destination hdfs://hs01:8020/User/hundsun/dap/Hive/hs_ods/ods_ses_acct_aSSuRe_scale/paRt_date=20210118/000000_0 at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.getHiveException(Hive.java:3449) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.getHiveException(Hive.java:3405) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.handlePoolException(Hive.java:3422) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.MOVefile(Hive.java:3367) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.ReplACEfiles(Hive.java:3697) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.loadPaRtITionInteRnal(Hive.java:1614) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.loadPaRtITion(Hive.java:1525) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.Metadata.Hive.loadPaRtITion(Hive.java:1489) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.exec.MOVeTask.execute(MOVeTask.java:501) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.exec.Task.executeTask(Task.java:199) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.exec.TaskRunneR.RunSequential(TaskRunneR.java:97) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql.DRiveR.launchTask(DRiveR.java:2200) ~[Hive-exec-2.1.1-cdh6.3.2.jaR:2.1.1-cdh6.3.2] at oRg.Apache.hadoop.Hive.ql