大数据集群各种报错及解决方案

一、启动hive报错

[root@master sbin]# hive
Hive Session ID = 991ccabe-96b4-4fae-8b1c-ac2856ab182e

Logging initialized using configuration in jar:file:/root/soft/hive/apache-hive-3.1.3-bin/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /tmp/hive/root/991ccabe-96b4-4fae-8b1c-ac2856ab182e. Name node is in safe mode.
The reported blocks 822 has reached the threshold 0.9990 of total blocks 822. The minimum number of live datanodes is not required. In safe mode extension. Safe mode will be turned off automatically in 6 seconds. NamenodeHostName:master
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.newSafemodeException(FSNamesystem.java:1570)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1557)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3406)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1161)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:739)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)

        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:651)
        at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:591)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /tmp/hive/root/991ccabe-96b4-4fae-8b1c-ac2856ab182e. Name node is in safe mode.
The reported blocks 822 has reached the threshold 0.9990 of total blocks 822. The minimum number of live datanodes is not required. In safe mode extension. Safe mode will be turned off automatically in 6 seconds. NamenodeHostName:master
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.newSafemodeException(FSNamesystem.java:1570)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1557)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3406)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1161)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:739)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
        at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2492)
        at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2466)
        at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1467)
        at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1464)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1481)
        at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1456)
        at org.apache.hadoop.hive.ql.session.SessionState.createPath(SessionState.java:786)
        at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:721)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:627)
        ... 9 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /tmp/hive/root/991ccabe-96b4-4fae-8b1c-ac2856ab182e. Name node is in safe mode.
The reported blocks 822 has reached the threshold 0.9990 of total blocks 822. The minimum number of live datanodes is not required. In safe mode extension. Safe mode will be turned off automatically in 6 seconds. NamenodeHostName:master
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.newSafemodeException(FSNamesystem.java:1570)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1557)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3406)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1161)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:739)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)

        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562)
        at org.apache.hadoop.ipc.Client.call(Client.java:1508)
        at org.apache.hadoop.ipc.Client.call(Client.java:1405)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:234)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:119)
        at com.sun.proxy.$Proxy28.mkdirs(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:673)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
        at com.sun.proxy.$Proxy29.mkdirs(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2490)
        ... 18 more

问题原因:
这个错误信息是Hadoop分布式文件系统(HDFS)中常见的问题,当你尝试在Hive或其他Hadoop应用程序中创建目录时遇到。具体来说,错误原因是NameNode(HDFS的元数据存储和管理器)正处于安全模式(Safe Mode)。安全模式是一种HDFS启动期间或在某些情况下为了维护数据完整性而执行的保护机制。

安全模式下,NameNode不会接受新的写入请求,包括创建新的目录或文件,以防止数据损坏。当出现以下情况时,NameNode可能会进入安全模式:

数据节点报告重大错误,可能导致数据不一致。
名称节点的目录结构发生变化,需要重新平衡数据块。
名称节点进行维护操作,如数据校验、备份或恢复。

为了解决这个问题,你可以尝试以下步骤:

等待:如果NameNode正在自动处理某个临时性的任务,可能很快就会退出安全模式。你可以检查集群的日志,确认是否有正在进行的任务。
重启NameNode:如果安全模式已经持续一段时间,或者你知道没有正在进行的操作,你可以尝试重启NameNode。但在生产环境中,这可能需要谨慎操作,并确保其他服务已经关闭或者有正确的备份。
手动解除安全模式:如果你有权限,可以通过命令行工具(如hadoop dfsadmin -safemode leave)来手动解除安全模式。但在此之前,要确保所有必要的检查都已完成。
检查集群健康:确保所有数据节点正常运行,没有异常退出,没有丢失的数据,以及数据块的均衡分配。

在执行任何操作之前,建议查阅集群的官方文档或联系系统管理员,以确保操作的正确性和安全性。

解决方案:

hadoop dfsadmin -safemode leave

二、Java代码连接远程hive报错

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/thrift/protocol/TProtocol
	at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:94)
	at java.sql.DriverManager.getConnection(DriverManager.java:664)
	at java.sql.DriverManager.getConnection(DriverManager.java:208)
	at com.example.demo.controller.TestController.main(TestController.java:39)
Caused by: java.lang.ClassNotFoundException: org.apache.thrift.protocol.TProtocol
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	... 4 more

初始化hive的时候报错
执行命令:./schematool -initSchema -dbType mysql

[root@master bin]# ./schematool -initSchema -dbType mysql
Metastore connection URL:        jdbc:mysql://127.0.0.1:3306/hive_metadata?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useSSL=false
Metastore Connection Driver :    com.mysql.cj.jdbc.Driver
Metastore connection User:       root
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version.
Underlying cause: java.sql.SQLNonTransientConnectionException : Public Key Retrieval is not allowed
SQL Error code: 0
Use --verbose for detailed stacktrace.
*** schemaTool failed ***

我是修改了MySQL的密码,但是MySQL还没有重新启动登录
重新启动登录后再执行这条命令后就没有问题了

java代码连接hive,操作insert语句向分区表中插入数据报错

org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException [Error 10096]: Dynamic partition strict mode requires at least one static partition column. To turn this off set hive.exec.dynamic.partition.mode=nonstrict
	at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:300)
	at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:286)
	at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:324)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:265)
	at org.apache.hive.jdbc.HiveStatement.executeUpdate(HiveStatement.java:511)
	at com.example.demo.controller.TestController.main(TestController.java:48)
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException [Error 10096]: Dynamic partition strict mode requires at least one static partition column. To turn this off set hive.exec.dynamic.partition.mode=nonstrict
	at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:199)
	at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:260)
	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:527)
	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:312)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:562)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.SemanticException:Dynamic partition strict mode requires at least one static partition column. To turn this off set hive.exec.dynamic.partition.mode=nonstrict
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer$TableSpec.<init>(BaseSemanticAnalyzer.java:1563)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer$TableSpec.<init>(BaseSemanticAnalyzer.java:1427)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2257)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2075)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:12033)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12129)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:659)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1826)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1773)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1768)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
	... 15 more

原因:
解决方案:在hive的终端下执行下面的命令

set hive.exec.dynamic.partition.mode=nonstrict

普通表向分区表中同步数据

hive> insert into Hive_dsj_6c_jky_yzhe_bigdata.ckxcsj PARTITION (SFDM,RQ) (select ID,CKRQJSJ,SJCPH,CPYS,CXDM,CZDM,ZS,CKZBH,CKZMC,RKZBH,RKZMC,RKRQJSJ,JFZLCS,SFKFSSFZ,ZFLX,TXSFGS,'aaa.zip','2024-06-11 10:11:19',SFDM,RQ    from Hive_dsj_6c_jky_yzhe_bigdata.cktx_11bj );
Query ID = root_20240607043952_75033e0d-fc0c-4662-986e-011a98265d1a
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1717720967189_0002, Tracking URL = http://master:8088/proxy/application_1717720967189_0002/
Kill Command = /home/wuzhanxi/soft/hadoop/hadoop-3.3.0/bin/mapred job  -kill job_1717720967189_0002
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2024-06-07 04:39:58,466 Stage-1 map = 0%,  reduce = 0%
2024-06-07 04:40:06,789 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_1717720967189_0002 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1717720967189_0002_m_000000 (and more) from job job_1717720967189_0002

Task with the most failures(4): 
-----
Task ID:
  task_1717720967189_0002_m_000000

URL:
  http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1717720967189_0002&tipid=task_1717720967189_0002_m_000000
-----
Diagnostic Messages for this Task:
Container launch failed for container_1717720967189_0002_01_000005 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:163)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:394)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

原因:我的yarn-xite.xml中没有配置相关内容,添加上解决了该问题

hive> insert into Hive_dsj_6c_jky_yzhe_bigdata.ckxcsj PARTITION (SFDM,RQ) (select ID,CKRQJSJ,SJCPH,CPYS,CXDM,CZDM,ZS,CKZBH,CKZMC,RKZBH,RKZMC,RKRQJSJ,JFZLCS,SFKFSSFZ,ZFLX,TXSFGS,'aaa.zip','2024-06-11 10:11:19',SFDM,RQ    from Hive_dsj_6c_jky_yzhe_bigdata.cktx_11bj );
FAILED: RuntimeException Cannot create staging directory 'hdfs://master:9000/tenant/Tenant_User_dsj_6c_jky_yzhe/Hive_dsj_6c_jky_yzhe_bigdata/ckxcsj/.hive-staging_hive_2024-06-07_04-53-06_746_8568341591675991282-1': Cannot create directory /tenant/Tenant_User_dsj_6c_jky_yzhe/Hive_dsj_6c_jky_yzhe_bigdata/ckxcsj/.hive-staging_hive_2024-06-07_04-53-06_746_8568341591675991282-1. Name node is in safe mode.
The reported blocks 12 has reached the threshold 0.9990 of total blocks 12. The minimum number of live datanodes is not required. In safe mode extension. Safe mode will be turned off automatically in 6 seconds. NamenodeHostName:master
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.newSafemodeException(FSNamesystem.java:1570)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1557)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3406)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1161)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:739)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)

问题原因:

这个信息看起来是在描述Hadoop分布式文件系统(HDFS)中的"Safe Mode"功能。当Hadoop的NameNode(命名节点)处于安全模式(Safe Mode)时,它限制了对文件系统的写入操作,确保系统的元数据一致性。在安全模式中,HDFS不允许新的数据块被创建,而是只进行读取和复制已存在的数据。

“The minimum number of live datanodes is not required” 表示,在安全模式下,并不需要满足最低数量的活跃DataNode(数据节点),因为此时主要关注的是数据完整性,而非数据写入。

“Safe mode will be turned off automatically in 6 seconds” 指的是NameNode计划在6秒后自动退出安全模式,条件可能包括检测到足够的DataNodes可用,或者进行了必要的数据同步。

“NamenodeHostName: master” 这部分提到了当前的NameNode服务运行在名为"master"的主机上。

如果你对HDFS的安全模式有更具体的问题,比如它的触发条件、退出条件、影响或如何手动退出安全模式等,可以继续提问。

过6秒钟再去执行这个插入SQL,发现已经没有问题了

相关推荐

  1. 数据各种解决方案

    2024-06-15 13:18:03       8 阅读
  2. centos安装paddlespeech各种解决方案

    2024-06-15 13:18:03       13 阅读

最近更新

  1. TCP协议是安全的吗?

    2024-06-15 13:18:03       18 阅读
  2. 阿里云服务器执行yum,一直下载docker-ce-stable失败

    2024-06-15 13:18:03       19 阅读
  3. 【Python教程】压缩PDF文件大小

    2024-06-15 13:18:03       18 阅读
  4. 通过文章id递归查询所有评论(xml)

    2024-06-15 13:18:03       20 阅读

热门阅读

  1. [whl]树莓派armv7l文件onnx的whl所有下载地址汇总

    2024-06-15 13:18:03       8 阅读
  2. 前端面经总结、学习【2023秋招】

    2024-06-15 13:18:03       11 阅读
  3. Vue实现excel导出,不请求后端

    2024-06-15 13:18:03       8 阅读
  4. YoloV8改进策略:卷积篇|Kan行天下之FastKANConv

    2024-06-15 13:18:03       13 阅读