大叔问题定位分享(20)hdfs文件create写入正常,append写入报错
2023-09-14 09:00:08 时间
最近在hdfs写文件的时候发现一个问题,create写入正常,append写入报错,每次都能重现,代码示例如下:
FileSystem fs = FileSystem.get(conf); OutputStream out = fs.create(file); IOUtils.copyBytes(in, out, 4096, true); //正常 out = fs.append(file); IOUtils.copyBytes(in, out, 4096, true); //报错
通过hdfs fsck命令检查出问题的文件,发现只有一个副本,难道是因为这个?
看FileSystem.append执行过程:
org.apache.hadoop.fs.FileSystem
public abstract FSDataOutputStream append(Path var1, int var2, Progressable var3) throws IOException;
实现类在这里:
org.apache.hadoop.hdfs.DistributedFileSystem
public FSDataOutputStream append(Path f, final int bufferSize, final Progressable progress) throws IOException { this.statistics.incrementWriteOps(1); Path absF = this.fixRelativePart(f); return (FSDataOutputStream)(new FileSystemLinkResolver<FSDataOutputStream>() { public FSDataOutputStream doCall(Path p) throws IOException, UnresolvedLinkException { return DistributedFileSystem.this.dfs.append(DistributedFileSystem.this.getPathName(p), bufferSize, progress, DistributedFileSystem.this.statistics); } public FSDataOutputStream next(FileSystem fs, Path p) throws IOException { return fs.append(p, bufferSize); } }).resolve(this, absF); }
这里会调用DFSClient.append方法
org.apache.hadoop.hdfs.DFSClient
private DFSOutputStream append(String src, int buffersize, Progressable progress) throws IOException { this.checkOpen(); DFSOutputStream result = this.callAppend(src, buffersize, progress); this.beginFileLease(result.getFileId(), result); return result; } private DFSOutputStream callAppend(String src, int buffersize, Progressable progress) throws IOException { LocatedBlock lastBlock = null; try { lastBlock = this.namenode.append(src, this.clientName); } catch (RemoteException var6) { throw var6.unwrapRemoteException(new Class[]{AccessControlException.class, FileNotFoundException.class, SafeModeException.class, DSQuotaExceededException.class, UnsupportedOperationException.class, UnresolvedPathException.class, SnapshotAccessControlException.class}); } HdfsFileStatus newStat = this.getFileInfo(src); return DFSOutputStream.newStreamForAppend(this, src, buffersize, progress, lastBlock, newStat, this.dfsClientConf.createChecksum()); }
DFSClient.append最终会调用NameNodeRpcServer的append方法
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer
public LocatedBlock append(String src, String clientName) throws IOException { this.checkNNStartup(); String clientMachine = getClientMachine(); if (stateChangeLog.isDebugEnabled()) { stateChangeLog.debug("*DIR* NameNode.append: file " + src + " for " + clientName + " at " + clientMachine); } this.namesystem.checkOperation(OperationCategory.WRITE); LocatedBlock info = this.namesystem.appendFile(src, clientName, clientMachine); this.metrics.incrFilesAppended(); return info; }
这里调用到FSNamesystem.append
org.apache.hadoop.hdfs.server.namenode.FSNamesystem
LocatedBlock appendFile(String src, String holder, String clientMachine) throws AccessControlException, SafeModeException, ... lb = this.appendFileInt(src, holder, clientMachine, cacheEntry != null); private LocatedBlock appendFileInt(String srcArg, String holder, String clientMachine, boolean logRetryCache) throws ... lb = this.appendFileInternal(pc, src, holder, clientMachine, logRetryCache); private LocatedBlock appendFileInternal(FSPermissionChecker pc, String src, String holder, String clientMachine, boolean logRetryCache) throws AccessControlException, UnresolvedLinkException, FileNotFoundException, IOException { assert this.hasWriteLock(); INodesInPath iip = this.dir.getINodesInPath4Write(src); INode inode = iip.getLastINode(); if (inode != null && inode.isDirectory()) { throw new FileAlreadyExistsException("Cannot append to directory " + src + "; already exists as a directory."); } else { if (this.isPermissionEnabled) { this.checkPathAccess(pc, src, FsAction.WRITE); } try { if (inode == null) { throw new FileNotFoundException("failed to append to non-existent file " + src + " for client " + clientMachine); } else { INodeFile myFile = INodeFile.valueOf(inode, src, true); BlockStoragePolicy lpPolicy = this.blockManager.getStoragePolicy("LAZY_PERSIST"); if (lpPolicy != null && lpPolicy.getId() == myFile.getStoragePolicyID()) { throw new UnsupportedOperationException("Cannot append to lazy persist file " + src); } else { this.recoverLeaseInternal(myFile, src, holder, clientMachine, false); myFile = INodeFile.valueOf(this.dir.getINode(src), src, true); BlockInfo lastBlock = myFile.getLastBlock(); if (lastBlock != null && lastBlock.isComplete() && !this.getBlockManager().isSufficientlyReplicated(lastBlock)) { throw new IOException("append: lastBlock=" + lastBlock + " of src=" + src + " is not sufficiently replicated yet."); } else { return this.prepareFileForWrite(src, iip, holder, clientMachine, true, logRetryCache); } } } } catch (IOException var11) { NameNode.stateChangeLog.warn("DIR* NameSystem.append: " + var11.getMessage()); throw var11; } } } public boolean isSufficientlyReplicated(BlockInfo b) { int replication = Math.min(this.minReplication, this.getDatanodeManager().getNumLiveDataNodes()); return this.countNodes(b).liveReplicas() >= replication; }
在append文件的时候,会首先取出这个文件最后一个block,然后会检查这个block是否满足副本要求,如果不满足就抛出异常,如果满足就准备写入;
看来原因确实是因为文件只有1个副本导致append报错,那为什么新建文件只有1个副本,后来找到原因是因为机架配置有问题导致的,详见 https://www.cnblogs.com/barneywill/p/10114504.html
相关文章
- nginx 判断访问文件或目录不存在rewrite
- [系统安全23]PE文件头中的重定位表
- [C++]怎么将.h和.cpp文件分别放在不同的目录
- idea 修改 jsp文件之后不生效问题
- Django普通文件上传
- 给你讲清楚什么是文件上传漏洞
- Java 动态生成 PDF 文件
- 如何在Linux下创建与解压zip, tar, tar.gz和tar.bz2文件
- Linux shell判断文件和文件夹是否存在
- 关于python文件操作
- Win7 文件加密存储操作后,如何在事后备份证书、秘钥
- SAP UI5 应用 manifest.json 文件里 Routes 数组元素的相对顺序,不可忽视的试读版
- SAP UI5 FileUploader 控件实现本地文件上传,接收服务器端的响应时遇到跨域访问错误的试读版
- Android Device Monitor中File Explorer的db文件如何导出到桌面
- Py之pandas:在表格文件中增加数据之for循环纵向/竖向非覆盖式增加数据到同一个csv文件内
- python将mat文件转为png
- eclipse中文件文件夹高速定位,打开文件所在文件夹,在资源管理器中查看
- vss 日文文件路径的名字在中文系统下乱码
- 文件切割
- 批量重命名文件
- Linux使用locate命令定位文件
- Android多线程文件下载器
- layui上传文件时如何加loading效果
- RHCSA之路----23、创建用于定位文件的脚本