Still have 1 requests outstanding when connection from slaveX/X.X.X.X:33202 is closed
ERROR TransportResponseHandler:
Still have 1 requests outstanding when connection from is closed
我加了如下的参数
spark.driver.memory=4g
spark.hadoop.mapreduce.input.fileinputformat.split.maxsize=134217728
spark.dynamicAllocation.initialExecutors=50
spark.yarn.executor.memoryOverhead=2048
spark.dynamicAllocation.maxExecutors=1000
spark.dynamicAllocation.tasksPerExecutorSlot=1
运行的程序其实逻辑上比较简单,只是从hive表里读取的数据量很大,差不多60+G,
并且需要将某些hive表读取到dirver节点上,用来获取每个executor上某些数据的映射值,
所以driver设定的资源较大。运行时抛出的异常信息,从网上查了下原因大致是服务器的
并发连接数超过了其承载量,服务器会将其中一些连接Down掉,这也就是说在运行spark程序时,过多的申请资源并发执行。
java.util.concurrent.RejectedExecutionException: Task scala.concurrent.impl.CallbackRunnable@3f9bf2d0 rejected from java.util.concurrent.ThreadPoolExecutor
spark 日志如上所示。还有一些其他日志,
18/06/11 18:40:43 ERROR TransportResponseHandler:
相关文章
- 【解决】librosa.load MP3返回空 或 报错Input signal length=0 is too small to resample from
- Maven项目报错:The type javax.validation.Payload cannot be resolved. It is indirectly referenced from req
- ORA-01189: file is from a different RESETLOGS than previous files ORACLE 报错 故障修复 远程处理
- ORA-01209: data file is from before the last RESETLOGS ORACLE 报错 故障修复 远程处理
- ORA-01287: file string is from a different database incarnation ORACLE 报错 故障修复 远程处理
- ORA-01576: The instance string is not enabled ORACLE 报错 故障修复 远程处理
- ORA-22618: attribute is a BAD NULL in the image handle ORACLE 报错 故障修复 远程处理
- ORA-25533: FAST_START_IO_TARGET or LOG_CHECKPOINT_INTERVAL is specified ORACLE 报错 故障修复 远程处理
- ORA-29555: Java source, class or resource is not allowed here ORACLE 报错 故障修复 远程处理
- ORA-29852: keyword IS is missing ORACLE 报错 故障修复 远程处理
- ORA-31691: The worker received message number string from the MCP, which is invalid. ORACLE 报错 故障修复 远程处理
- ORA-01209: data file is from before the last RESETLOGS ORACLE 报错 故障修复 远程处理
- ORA-01232: cannot start online backup – file string is being made read-only ORACLE 报错 故障修复 远程处理
- ORA-02038: define is not allowed for array type ORACLE 报错 故障修复 远程处理
- ORA-14052: partition-extended table name syntax is disallowed in this context ORACLE 报错 故障修复 远程处理
- ORA-14206: table is not subpartitioned by List or Range methods ORACLE 报错 故障修复 远程处理
- ORA-14609: Tablespace was not specified for the previous lob segments of column string in template but is specified for string ORACLE 报错 故障修复 远程处理
- ORA-15475: usage string is greater than string characters ORACLE 报错 故障修复 远程处理
- ORA-16417: Activation occurred after recovery from standby redo log files; a full database backup is required ORACLE 报错 故障修复 远程处理
- Mysql报错java.sql.SQLException:null,message from server:”Host ‘27,45,38,132’ is not allowed to connect详解数据库
- nullOracle中使用IS NULL代替NULL值(Oracle中is)
- MySQL实现数据库信息变更:From函数使用详解(mysql from())